ConcurrentModificationException when serializing LockedResourcesBuildAction#lockedResources

[dsilakov]

Jenkins and plugins versions report

Environment

Jenkins: 2.436
OS: Linux - 4.15.0-121-generic
Java: 11.0.21 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
Parameterized-Remote-Trigger:3.2.0
active-directory:2.34
allure-jenkins:3.29.0
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.2.3-1.0
authentication-tokens:1.53.v1c90fd9191a_b_
blueocean:1.27.9
blueocean-bitbucket-pipeline:1.27.9
blueocean-commons:1.27.9
blueocean-config:1.27.9
blueocean-core-js:1.27.9
blueocean-dashboard:1.27.9
blueocean-display-url:2.4.2
blueocean-events:1.27.9
blueocean-git-pipeline:1.27.9
blueocean-github-pipeline:1.27.9
blueocean-i18n:1.27.9
blueocean-jwt:1.27.9
blueocean-personalization:1.27.9
blueocean-pipeline-api-impl:1.27.9
blueocean-pipeline-editor:1.27.9
blueocean-pipeline-scm-api:1.27.9
blueocean-rest:1.27.9
blueocean-rest-impl:1.27.9
blueocean-web:1.27.9
bootstrap5-api:5.3.2-3
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1135.v8de8e7899051
build-name-setter:2.4.0
build-user-vars-plugin:1.9
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloud-stats:320.v96b_65297a_4b_b_
cloudbees-bitbucket-branch-source:856.v04c46c86f911
cloudbees-folder:6.858.v898218f3609d
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
config-file-provider:959.vcff671a_4518b_
configuration-as-code:1763.vb_fe9c1b_83f7b
credentials:1311.vcf0a_900b_37c2
credentials-binding:642.v737c34dea_6c2
data-tables-api:1.13.8-2
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-java-api:3.3.4-86.v39b_a_5ede342c
docker-plugin:1.5
durable-task:523.va_a_22cf15d5e0
echarts-api:5.4.3-2
email-ext:2.102
favorite:2.208.v91d65b_7792a_c
folder-properties:1.2.1
font-awesome-api:6.5.1-1
git:5.2.1
git-client:4.6.0
git-parameter:0.9.19
github:1.37.3.1
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1767.va_7d01ea_c7256
gitlab-api:5.3.0-91.v1f9a_fda_d654f
gitlab-branch-source:684.vea_fa_7c1e2fe3
gitlab-logo:1.1.2
gitlab-plugin:1.7.16
gson-api:2.10.1-3.vb_25b_599b_e4f8
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.32
http_request:1.18
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.3-372.v309620682326
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.9
jersey2-api:2.41-133.va_03323b_a_1396
jjwt-api:0.11.5-77.v646c772fddb_0
job-dsl:1.87
joda-time-api:2.12.5-5.v5495a_235fedf
jquery3-api:3.7.1-1
jsch:0.2.8-65.v052c39de79b_2
json-api:20231013-3.v20f3c247f2fe
json-path-api:2.8.0-5.v07cb_a_1ca_738c
junit:1252.vfc2e5efa_294f
kubernetes:4151.v6fa_f0fb_0b_4c9
kubernetes-client-api:6.8.1-224.vd388fca_4db_3b_
kubernetes-credentials:0.11
lockable-resources:1246.v28b_e4cc6fa_16
mailer:463.vedf8358e006b_
matrix-auth:3.2.1
matrix-project:818.v7eb_e657db_924
metrics:4.2.18-442.v02e107157925
mina-sshd-api-common:2.11.0-86.v836f585d47fa_
mina-sshd-api-core:2.11.0-86.v836f585d47fa_
multibranch-scan-webhook-trigger:1.0.11
okhttp-api:4.11.0-157.v6852a_a_fa_ec11
ownership:0.13.0
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:689.veec561a_dee13
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2151.ve32c9d209a_3f
pipeline-model-definition:2.2151.ve32c9d209a_3f
pipeline-model-extensions:2.2151.ve32c9d209a_3f
pipeline-rest-api:2.34
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2151.ve32c9d209a_3f
pipeline-stage-view:2.34
pipeline-utility-steps:2.16.0
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.6.0
prism-api:1.29.0-10
prometheus:2.4.1
pubsub-light:1.18
pyenv-pipeline:2.1.2
rebuild:330.v645b_7df10e2a_
role-strategy:689.v731678c3e0eb_
scm-api:683.vb_16722fb_b_80b_
script-security:1294.v99333c047434
snakeyaml-api:2.2-111.vc6598e30cc65
sse-gateway:1.26
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.945.v90744dcb_47cd
sshd:3.312.v1c601b_c83b_0e
structs:325.vcb_307d2a_2782
support-core:1366.v9d076592655d
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.133.vfb_8a_7b_9c5dd1
uno-choice:2.8.1
variant:60.v7290fc0eb_b_cd
workflow-api:1283.v99c10937efcb_
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3826.v3b_5707fe44da_
workflow-durable-task-step:1313.vcb_970b_d2a_fb_3
workflow-job:1385.vb_58b_86ea_fff1
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:865.v43e78cc44e0d

What Operating System are you using (both controller, and any agents involved in the problem)?

Debian 12

Reproduction steps

Run a pipeline consuming several dozens of lockable resources.

Several hundreds such pipelines a day often trigger such an issue

Expected Results

Pipeline works fine

Actual Results

From time to time, pipeline fails and we see the following in the end of console log:

all_2024-09-10_09.30.58.log-Also:   java.util.ConcurrentModificationException
all_2024-09-10_09.30.58.log-            at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
all_2024-09-10_09.30.58.log-            at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270)
all_2024-09-10_09.30.58.log:    Caused: java.lang.RuntimeException: Failed to serialize org.jenkins.plugins.lockableresources.actions.LockedResourcesBuildAction#lockedResources for class org.jenkins.plugins.lockableresources.actions.LockedResourcesBuildAction
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:87)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeBareItem(AbstractCollectionConverter.java:94)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:66)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeCompleteItem(AbstractCollectionConverter.java:81)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:75)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270)
all_2024-09-10_09.30.58.log-    Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 2bb0e54b-5e0a-4b52-a3d5-341f9a16bcbe
all_2024-09-10_09.30.58.log-    Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)

Unfortunately this breaks BlueOcean representation for the pipelines.

Anything else?

#597 was about the same problem but with older plugin version.

We updated to 1246.v28b_e4cc6fa_16 since we were able to update to it without touching any other plugins, so plugin version list from the description of this bug is still valid. This version should have all fixes mentioned in #597

Do you think that it makes sense to update some other plugins, as well? Or the reason still lies somewhere in lockable-resources itself?

Are you interested in contributing a fix?

No response

[mPokornyETM]

Oh no, pls not again. :-(
But good news, I found the same exception by fixing this PR #673

We use the changes over 3 mounts and does not find some side effects. When you agree, I can merge the PR (without review again :-( ) and you need just update the plugin.

Back to your question. This is failure in this plugin, and you does not need to update some other plugins.

It will be great, when you install this version on your Jenkins instance
https://ci.jenkins.io/job/Plugins/job/lockable-resources-plugin/job/PR-673/11/artifact/org/6wind/jenkins/lockable-resources/1329.v19c14cb_b_d796/
to verify, that ot works. In that case pls write some hit in the PR, that it is also verified by someone else (for transparency)

Anyway: just give me feadback and I will try to merge it into master, so our issues will be fixed ASAP ;-)

[dsilakov]

Ok, will give it a try, thanks

[dsilakov]

1329.v19c14cb_b_d796 has been installed. Running for 5 hours up to now without serialization issues or other problems, but I think it makes sense to monitor for longer time

[mPokornyETM]

1329.v19c14cb_b_d796 has been installed. Running for 5 hours up to now without serialization issues or other problems, but I think it makes sense to monitor for longer time

Let me know next week. When it works over few days without issues on your side, I will megre it into official master.

Thx for your feadback

[dsilakov]

So far so good, but we started to get a lot of messages about failures to save LR state:

Do you have any ideas what can cause this?

[mPokornyETM]

No, We does not save the LRM state. It cost too much time for us.
But I will try to check it.

[mPokornyETM]

I analysed the changes now. I does not see reason, why it might fail the save() function.
@dsilakov Can you check the persmission om your jenkins instance. Like '''ls -la''' Maybe somebody rewrite the file content and the standard locale Jenkins user has no permission to rewrite it now.

[dsilakov]

Yes, somebody constantly resets ownership of this file. If I change owner:group to jenkins:jenkins, the error goes away, however very soon the permissions are reset to root:jenkins. I'll try to figure out who can it be, but the thing is that we have the same root:jenkins permissions on other jenkins instances with older plugins and this doesn't lead to any problems. One more point is that in order to update LR, we had to update a bunch of other plugins, maybe some of that update influences our scenario. Will check. But as for original problem with serialization - likely yes, the new version solves it, never seen it again.

[mPokornyETM]

Great news, I will merge it now into master.