ci.jenkins.io no longer publishes incrementals

[MarkEWaite]

Jenkins and plugins versions report

Environment

Jenkins: 2.319.1
OS: Linux - 5.4.0-1064-azure
---
ace-editor:1.1
analysis-model-api:10.8.1
ansicolor:1.0.1
ant:1.13
antisamy-markup-formatter:2.5
apache-httpcomponents-client-4-api:4.5.13-1.0
authentication-tokens:1.4
aws-credentials:1.33
aws-java-sdk:1.12.101-300.vc09c7be9cb57
aws-java-sdk-cloudformation:1.12.101-300.vc09c7be9cb57
aws-java-sdk-codebuild:1.12.101-300.vc09c7be9cb57
aws-java-sdk-core:1.12.69
aws-java-sdk-ec2:1.12.101-300.vc09c7be9cb57
aws-java-sdk-ecr:1.12.101-300.vc09c7be9cb57
aws-java-sdk-ecs:1.12.101-300.vc09c7be9cb57
aws-java-sdk-elasticbeanstalk:1.12.101-300.vc09c7be9cb57
aws-java-sdk-iam:1.12.101-300.vc09c7be9cb57
aws-java-sdk-jmespath:1.12.69
aws-java-sdk-kms:1.12.69
aws-java-sdk-logs:1.12.101-300.vc09c7be9cb57
aws-java-sdk-minimal:1.12.101-300.vc09c7be9cb57
aws-java-sdk-s3:1.12.69
aws-java-sdk-ssm:1.12.101-300.vc09c7be9cb57
aws-java-sdk-sts:1.12.69
azure-container-agents:215.v9ca34d6f145b
azure-credentials:198.vf9c2fdfde55c
azure-sdk:70.v63f6a95999a7
azure-vm-agents:799.va4c741108611
basic-branch-build-strategies:1.3.2
beer:1.3
blueocean:1.25.2
blueocean-autofavorite:1.2.4
blueocean-bitbucket-pipeline:1.25.2
blueocean-commons:1.25.2
blueocean-config:1.25.2
blueocean-core-js:1.25.2
blueocean-dashboard:1.25.2
blueocean-display-url:2.4.1
blueocean-events:1.25.2
blueocean-git-pipeline:1.25.2
blueocean-github-pipeline:1.25.2
blueocean-i18n:1.25.2
blueocean-jira:1.25.2
blueocean-jwt:1.25.2
blueocean-personalization:1.25.2
blueocean-pipeline-api-impl:1.25.2
blueocean-pipeline-editor:1.25.2
blueocean-pipeline-scm-api:1.25.2
blueocean-rest:1.25.2
blueocean-rest-impl:1.25.2
blueocean-web:1.25.2
bootstrap4-api:4.6.0-3
bootstrap5-api:5.1.3-4
bouncycastle-api:2.25
branch-api:2.7.0
build-timeout:1.20
buildtriggerbadge:2.11
caffeine-api:2.9.2-29.v717aac953ff3
checks-api:1.7.2
cloud-stats:0.27
cloudbees-bitbucket-branch-source:734.v2f848c5e6ea2
cloudbees-folder:6.17
code-coverage-api:2.0.4
command-launcher:1.6
conditional-buildstep:1.4.1
configuration-as-code:1.55
copyartifact:1.46.2
credentials:1055.v1346ba467ba1
credentials-binding:1.27
cvs:2.19
dark-theme:155.v497c78bbdbb3
data-tables-api:1.11.3-6
disable-github-multibranch-status:1.2
display-url-api:2.3.5
docker-commons:1.17
docker-workflow:1.26
durable-task:493.v195aefbb0ff2
ec2:1.66
echarts-api:5.2.2-2
embeddable-build-status:2.0.3
extended-read-permission:3.2
external-monitor-job:1.7
favorite:2.3.3
font-awesome-api:5.15.4-5
forensics-api:1.7.0
git:4.10.1
git-client:3.10.0
git-forensics:1.3.0
git-server:1.10
github:1.34.1
github-api:1.301-378.v9807bd746da5
github-branch-source:2.11.3
github-checks:1.0.14
groovy:2.4
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-1.0
htmlpublisher:1.28
http_request:1.12
jackson2-api:2.13.1-242.v1a45bad25ceb
jacoco:3.3.0
javadoc:1.6
jaxb:2.3.0.1
jdk-tool:1.5
jenkins-design-language:1.25.2
jira:3.6
jjwt-api:0.11.2-9.c8b45b8bb173
jobConfigHistory:2.30
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.6.0-2
jsch:0.1.55.2
junit:1.53
junit-attachments:1.7
junit-realtime-test-reporter:72.v203572adb123
keyboard-shortcuts-plugin:1.4
kubernetes:1.31.1
kubernetes-client-api:5.10.1-171.vaa0774fb8c20
kubernetes-credentials:0.9.0
ldap:2.7
lockable-resources:2.13
mailer:1.34
mapdb-api:1.0.9.0
matrix-auth:3.0
matrix-project:1.19
maven-plugin:3.16
mercurial:2.16
metrics:4.0.2.8
momentjs:1.1.1
node-iterator-api:1.5.1
okhttp-api:4.9.3-105.vb96869f8ac3a
pam-auth:1.6.1
parallel-test-executor:1.13
parameterized-trigger:2.43
pipeline-build-step:2.15
pipeline-github-lib:1.0
pipeline-githubnotify-step:1.0.5
pipeline-graph-analysis:1.12
pipeline-graph-view:47.v91726240f7a6
pipeline-input-step:427.va6441fa17010
pipeline-milestone-step:1.3.2
pipeline-model-api:1.9.3
pipeline-model-definition:1.9.3
pipeline-model-extensions:1.9.3
pipeline-rest-api:2.20
pipeline-stage-step:2.5
pipeline-stage-tags-metadata:1.9.3
pipeline-stage-view:2.20
pipeline-utility-steps:2.11.0
plain-credentials:1.7
plugin-util-api:2.8.0
popper-api:1.16.1-2
popper2-api:2.11.0-1
prism-api:1.25.0-1
pubsub-light:1.16
run-condition:1.5
scm-api:2.6.5
script-security:1118.vba21ca2e3286
snakeyaml-api:1.29.1
sse-gateway:1.24
ssh-agent:1.23
ssh-credentials:1.19
ssh-slaves:1.33.0
sshd:3.1.0
structs:308.v852b473a2b8c
subversion:2.15.1
support-core:2.79
theme-manager:0.6
throttle-concurrents:2.5
timestamper:1.15
token-macro:267.vcdaea6462991
toolenv:1.2
translation:1.16
trilead-api:1.0.13
variant:1.4
warnings-ng:9.10.2
windows-azure-storage:365.vf41653c43b01
windows-slaves:1.8
workflow-aggregator:2.6
workflow-api:1108.v57edf648f5d4
workflow-basic-steps:2.24
workflow-cps:2648.va9433432b33c
workflow-cps-global-lib:552.vd9cc05b8a2e1
workflow-durable-task-step:1107.v5dab75aaccbd
workflow-job:1145.v7f2433caa07f
workflow-multibranch:2.26
workflow-scm-step:2.13
workflow-step-api:615.vb09dac339255
workflow-support:804.vba10a18a1476  

What Operating System are you using (both controller, and any agents involved in the problem)?

Ubuntu 18.04 for controller and for agents

Reproduction steps

1. Enable incrementals for a Jenkins plugin
2. Build the Jenkins plugin on ci.jenkins.io with the buildPlugin() pipeline library
3. Configure a plugins.txt file in a Dockerfile to use the plugin that has been built. In my case, I used

git-client:incrementals;org.jenkins-ci.plugins;3.10.2-rc3007.0409ea_938a_3a_

4. Attempt to download the plugin with plugin installation manager

Expected Results

Plugin downloaded and available for use

Actual Results

Plugin download fails because the incremental build was not uploaded to repos.jenkins-ci.org.

Build log reports:

10:02:25  [Pipeline] {
10:02:25  [Pipeline] httpRequest
10:02:25  HttpMethod: POST
10:02:25  URL: https://incrementals.jenkins.io/
10:02:25  Content-Type: application/json
10:02:25  Authorization: *****
10:02:25  Sending request to url: https://incrementals.jenkins.io/
10:02:26  Response Code: HTTP/1.1 200 OK
10:02:26  Response: 
10:02:26  Skipping deployment as no artifacts were found with the expected path, typically due to a PR merge build not up to date with its base branch: https://ci.jenkins.io/job/Plugins/job/git-client-plugin/job/master/728/artifact/**/*0409ea938a3a*/*0409ea938a3a*/*zip*/archive.zip
10:02:26  
10:02:26  Success: Status code 200 is in the accepted range: 100:599
10:02:26  [Pipeline] }

Anything else?

Likely needs a change in the argument to the archiveArtifacts task in the pipeline library used on ci.jenkins.io.

[jglick]

jenkins-infra/incrementals-publisher#20 should have fixed this. Not sure what is going on with the deployment. jenkins-infra/helm-charts#57? jenkins-infra/helm-charts#67? @timja or @jetersen do you know?

[timja]

This was reported elsewhere where I gave more detail but basically we're hit by https://issues.jenkins.io/browse/JENKINS-67474 which has caused complete blockage of infra builds preventing updates.

It's being worked on in:

• fix: use jnlp inbound-agent image as source, bump aws-cli version and add yamllint jenkins-infra/docker-helmfile#45
• Extend from inbound-agent jenkins-infra/docker-builder#30
• fix(kubernetes-pod agents) switch to single-container pod pattern jenkins-infra/pipeline-library#273

Getting close atm I think, mostly @dduportal and @lemeurherve working on it

Current state from what I can tell is the parallelDockerUpdatecli is broken from pipeline-library, can be seen in build logs for jenkins-infra/docker-helmfile#45

[dduportal]

For information, the branch master of this project is now building successfully on infra.ci.jenkins.io: https://infra.ci.jenkins.io/job/incrementals-publisher/job/master/ .

Last mile for us, before being able to check ci.jenkins.io is to ensure that our kubernetes management works as expected.

[dduportal]

Kubernetes management is now operational "again" and has detected the changes on the incremental application:

• https://infra.ci.jenkins.io/job/helm-charts/job/main/2/console is now working as expected (updatecli process for detecting applications update and open PRs to update their helm charts - https://github.com/jenkins-infra/helm-charts/blob/main/Jenkinsfile_k8s)
• [updatecli] Bump incremental-publisher docker image and helm chart versions jenkins-infra/helm-charts#67 had been updated and builds successfully. Approval & merge in progress.
• In a few minutes, https://github.com/jenkins-infra/kubernetes-management will automatically pick the change and update.

[dduportal]

One last mile was required: jenkins-infra/kubernetes-management#1886.

The last known version of the incrementalm-publisher (v1.4.2) is deployed successfully:

Successfully pulled image "jenkinsciinfra/incrementals-publisher:v1.4.2" in 25.898692355s

=> Can you confirm that incremental publishing is working again as expected?

[MarkEWaite]

=> Can you confirm that incremental publishing is working again as expected?

https://ci.jenkins.io/job/Plugins/job/git-client-plugin/job/master/735/console confirms that incremental plugin publishing is working. The build results are deployed to repos.jenkins-ci.org.

I used git-client:incrementals;org.jenkins-ci.plugins;3.11.1-rc3016.2b_fb_197d7b_9c in my plugins.txt file and the plugin installation manager downloaded that plugin version from repos.jenkins-ci.org.