Make Xtext Sign and Deploy process more future proof

[cdietrich]

our current sign and deploy process is a big obscure mess nobody who cares anymore understands and that is full of tech. debt

• Switch Jenkinsfiles to Centos 8 #2057
• https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/1226
• Does not work with Gradle 7 xtext/publishing#24
• [WARNING] label option is deprecated. To use a static pod template, use the 'inheritFrom' option. #1987

we use a stoneage gradle version and old centos and old java
that worries me a lot.

[cdietrich]

@kthoms @szarnekow @nprediger this is something where i really need help

[kthoms]

When changing to centos-8 the build fails with

14:29:24  > Task :signArchivesXtextCore FAILED
14:29:24  :signArchivesXtextCore (Thread[Daemon worker,5,main]) completed. Took 0.022 secs.
14:29:24  
14:29:24  FAILURE: Build failed with an exception.
14:29:24  
14:29:24  * What went wrong:
14:29:24  Execution failed for task ':signArchivesXtextCore'.
14:29:24  > Unable to retrieve secret key from key ring file '/home/jenkins/.gnupg/secring.gpg' as it does not exist

[kthoms]

Working build: https://ci.eclipse.org/xtext/job/releng/job/sign-and-deploy/1535/consoleFull

03:50:20  Xtext version on branch master is 2.28.0-SNAPSHOT
03:50:20  + gpg --batch --import ****
03:50:20  gpg: directory `/home/jenkins/.gnupg' created
03:50:20  gpg: new configuration file `/home/jenkins/.gnupg/gpg.conf' created
03:50:20  gpg: WARNING: options in `/home/jenkins/.gnupg/gpg.conf' are not yet active during this run
03:50:20  gpg: keyring `/home/jenkins/.gnupg/secring.gpg' created
03:50:20  gpg: keyring `/home/jenkins/.gnupg/pubring.gpg' created
03:50:20  gpg: key 084853F7: secret key imported
03:50:20  gpg: /home/jenkins/.gnupg/trustdb.gpg: trustdb created
03:50:20  gpg: key 084853F7: public key "Eclipse Xtext Project <[email protected]>" imported
03:50:20  gpg: Total number processed: 1
03:50:20  gpg:               imported: 1  (RSA: 1)
03:50:20  gpg:       secret keys read: 1
03:50:20  gpg:   secret keys imported: 1
03:50:20  ++ gpg --list-keys --with-colons

Failing build: https://ci.eclipse.org/xtext/job/releng/job/sign-and-deploy/1536/consoleFull

14:28:06  Xtext version on branch master is 2.28.0-SNAPSHOT
14:28:06  + gpg --batch --import ****
14:28:06  gpg: directory '/home/jenkins/.gnupg' created
14:28:06  gpg: keybox '/home/jenkins/.gnupg/pubring.kbx' created
14:28:06  gpg: /home/jenkins/.gnupg/trustdb.gpg: trustdb created
14:28:06  gpg: key DA31DD9C084853F7: public key "Eclipse Xtext Project <[email protected]>" imported
14:28:06  gpg: To migrate 'secring.gpg', with each smartcard, run: gpg --card-status
14:28:06  gpg: key DA31DD9C084853F7: secret key imported
14:28:06  gpg: Total number processed: 1
14:28:06  gpg:               imported: 1
14:28:06  gpg:       secret keys read: 1
14:28:06  gpg:   secret keys imported: 1

[kthoms]

GPG versions:
centos-7: gpg (GnuPG) 2.0.22
centos-8: gpg (GnuPG) 2.2.20

With GPG 2.2 a new file format was introduced for storuing the GPG keyring.

[kthoms]

https://docs.gradle.org/current/userguide/signing_plugin.html:

Since gpg 2.1, you need to export the keys with command gpg --keyring secring.gpg --export-secret-keys > ~/.gnupg/secring.gpg

[kthoms]

15:15:35  + gpg --keyring secring.gpg --export-secret-keys
15:15:35  gpg: starting migration from earlier GnuPG versions
15:15:35  gpg: porting secret keys from '/home/jenkins/.gnupg/secring.gpg' to gpg-agent
15:15:35  gpg: migration succeeded
15:15:35  gpg: key 07770761964E4F3E397222D29492D728898040B0: error receiving key from agent: Inappropriate ioctl for device - skipped

[cdietrich]

@kthoms yes this is where i was stuck
https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/1226

[kthoms]

We tried a gazillion of permutations. Next step is to disable the GPG magic in the publishing plugin and do this by shell scripting.
@nprediger and I have set a follow-up for June 22th

[kthoms]

We are thinking to do the Eclipse signing of the jars already within the regular build on the master branch. This would take off more magic from the publishing plugin.

[cdietrich]

I assume then we also can’t use gradle to deploy too

[cdietrich]

@kthoms @nprediger i have found this one
https://github.com/xtext/publishing/blob/343c89fb465ba40f8cbfd0652752d47a845718a0/buildSrc/src/main/java/io/typefox/publishing/MavenPublishing.xtend#L331
here PublishingPlugin.SIGNING_PASSWORD
is set. but the gpgSign() thing seems not to use
signing.password but signing.gnupg.passphrase instead (the keyname also needs to be
signing.gnupg.keyName

[cdietrich]

@kthoms @nprediger i got this one running
https://github.com/xtext/gradle-sign-test
https://ci.eclipse.org/xtext/job/releng/job/gradle-sign-test/job/main/

[kthoms]

@nprediger and I are continuing today.

Seems that we have some progress now.

14:45:33  Invoking gpg2 with arguments: [--local-user, D1AE0CFD, --pinentry-mode=loopback, --passphrase, ****, --no-tty, --batch, --detach-sign]
14:45:33  Starting process 'command 'gpg2''. Working directory: /home/jenkins/agent/workspace/releng/sign-and-deploy/publishing Command: gpg2 --local-user D1AE0CFD --pinentry-mode=loopback --passphrase **** --no-tty --batch --detach-sign
14:45:33  Successfully started process 'command 'gpg2''
14:45:33  Invoking gpg2 with arguments: [--local-user, D1AE0CFD, --pinentry-mode=loopback, --passphrase, ****, --no-tty, --batch, --detach-sign]
14:45:33  Starting process 'command 'gpg2''. Working directory: /home/jenkins/agent/workspace/releng/sign-and-deploy/publishing Command: gpg2 --local-user D1AE0CFD --pinentry-mode=loopback --passphrase **** --no-tty --batch --detach-sign
14:45:33  Successfully started process 'command 'gpg2''

...

14:46:35  Publishing to repository 'ossrh' (https://oss.sonatype.org/content/repositories/snapshots/)
14:46:35  Uploading maven-metadata.xml to /content/repositories/snapshots/org/eclipse/xtext/org.eclipse.xtext.builder.standalone/2.28.0-SNAPSHOT/maven-metadata.xml
14:46:35  Uploading org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26.pom to /content/repositories/snapshots/org/eclipse/xtext/org.eclipse.xtext.builder.standalone/2.28.0-SNAPSHOT/org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26.pom
14:46:35  Uploading org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26-sources.jar to /content/repositories/snapshots/org/eclipse/xtext/org.eclipse.xtext.builder.standalone/2.28.0-SNAPSHOT/org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26-sources.jar
14:46:35  Uploading org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26-javadoc.jar to /content/repositories/snapshots/org/eclipse/xtext/org.eclipse.xtext.builder.standalone/2.28.0-SNAPSHOT/org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26-javadoc.jar
14:46:35  Uploading org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26-sources.jar.asc to /content/repositories/snapshots/org/eclipse/xtext/org.eclipse.xtext.builder.standalone/2.28.0-SNAPSHOT/org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26-sources.jar.asc
14:46:35  Uploading org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26.jar to /content/repositories/snapshots/org/eclipse/xtext/org.eclipse.xtext.builder.standalone/2.28.0-SNAPSHOT/org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26.jar
14:46:35  Uploading org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26-javadoc.jar.asc to /content/repositories/snapshots/org/eclipse/xtext/org.eclipse.xtext.builder.standalone/2.28.0-SNAPSHOT/org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26-javadoc.jar.asc
14:46:35  Uploading org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26.pom.asc to /content/repositories/snapshots/org/eclipse/xtext/org.eclipse.xtext.builder.standalone/2.28.0-SNAPSHOT/org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26.pom.asc
14:46:35  Uploading org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26.jar.asc to /content/repositories/snapshots/org/eclipse/xtext/org.eclipse.xtext.builder.standalone/2.28.0-SNAPSHOT/org.eclipse.xtext.builder.standalone-2.28.0-20220622.124415-26.jar.asc
14:46:35  Uploading maven-metadata.xml to /content/repositories/snapshots/org/eclipse/xtext/org.eclipse.xtext.builder.standalone/maven-metadata.xml
14:46:35  :publishOrgeclipsextextbuilderstandalonePublicationToOssrhRepository (Thread[Daemon worker,5,main]) completed. Took 1.699 secs.

[kthoms]

Based on the work from @cdietrich we have added the GPG credentials to the pipeline. This lead us to the publishing plugin since it then required the possibility to pass the passphrase as parameter. Further it then complained:

No value has been specified for property 'signatory.keyId'

We have seen that the property ''signing.gnupg.keyName' should be set to the value of 'signing.keyId'.

We did some adjustments on the publishing plugin to handle the additional properties. Further it seems that we have to use useGpgCmd() for the signing plugin. Without it did not work.

[LorenzoBettini]

@cdietrich I think we can close this one?

[cdietrich]

yes