Target Platform for 3rd parties: Orbit vs Maven

[cdietrich]

Target Platform for 3rd parties: Orbit vs Maven
what happens if orbit is discontinued

problems / knowledge gaps for maven dependencies in target

• oomph: to my knowledge oomph cannot do it and platform cheats around it using a "private orbit" / using previous built p2 repo
• tycho 2.7.5: we currently dont use a new enhough tycho cause tycho has problems with pom dependencies packages to target platforms but not available in repo Milestone does not build on Jenkins xtext-eclipse#1931
• tycho 3.0 requires java 17 and thus toolchain setup. this might complicate stuff.
• we build parts with gradle then build features with tycho in xtext-umbrella and also repackage orbit stuff there https://github.com/eclipse/xtext-umbrella/blob/master/releng/org.eclipse.xtext.sdk.p2-repository/category.xml (or inlude them in features). this repo is then signed and deployed via https://github.com/eclipse/xtext-umbrella/blob/master/releng/jenkins/sign-and-deploy/Jenkinsfile which uses https://github.com/xtext/publishing/ to sign stuff in the p2 repo (and the maven artifacts, both curling the jarsigner serivce to my knowledge). unclear what to do with the maven dependencies there (the ones we repackage) and how to separate the two things.
  (maybe https://docs.google.com/document/d/1dl10ia092X5hN1qfKoHYvriCNM-iBqiOkfjnntxaBbk/edit ?!? - is there a better documentation or example somewhere?)
• is it possible to manage the version of the maven dependencies using a dependencyManagent config / bom ?
• in customer projects i have seen terrible build times using this feature
• ..... 1000 problems we dont see yet

[cdietrich]

cc @merks @kthoms this collects some of the knowledge gaps and problems i see with "using maven artifacts in target"

[merks]

Note that Oomph now does support including a normal *.target into a Targlet container and that target can contain Maven dependencies. This work was funding by https://www.diamond.ac.uk/

The approach I described in the forum will allow Maven dependencies in a centrally managed *.target file (along with distributed managed additional *.target files from SimRel projects) to be used to generate a normal p2 repository that can be consumed in the normal way exactly as you are currently consuming from Orbit. Mind you, the artifacts are PGP-signed, not jar-signed.

Will that address all your concerns?

[cdietrich]

@merks with orbit i had that impression that projects are "encouraged" to repackage their dependencies into their own repo. now when i will repackage the pgp signed stuff from the "new pgp signed orbit", will these become pgp signed automatically (just copy over the original signing info) or will they be unsigned and thus we have to care about signing again (point 4 in my list)

will do some experiments on this

[merks]

The PGP key and signature are properties on the artifact, e.g.,

    <artifact classifier='osgi.bundle' id='com.ibm.icu' version='72.1.0'>
      <properties size='14'>
        <property name='maven-groupId' value='com.ibm.icu'/>
        <property name='maven-artifactId' value='icu4j'/>
        <property name='maven-version' value='72.1'/>
        <property name='maven-repository' value='eclipse.maven.central.mirror'/>
        <property name='maven-type' value='jar'/>
        <property name='download.checksum.sha-1' value='bc9057df4b5efddf7f6d1880bf7f3399f4ce5633'/>
        <property name='download.size' value='14386028'/>
        <property name='artifact.size' value='14386028'/>
        <property name='download.md5' value='b98735702f497dd482638bedc16a1c09'/>
        <property name='download.checksum.sha-512' value='48130eb346a5bb5bdaff867e5231808a3cd03d1876d8e6b7c501336df6992e912f7c53456dc72b673ad3272f3b54b414343eea8a1118d01d0e517403cab3e324'/>
        <property name='download.checksum.md5' value='b98735702f497dd482638bedc16a1c09'/>
        <property name='download.checksum.sha-256' value='3df572b240a68d13b5cd778ad2393e885d26411434cd8f098ac5987ea2e64ce3'/>
        <property name='pgp.signatures' value='-----BEGIN PGP SIGNATURE-----&#10;&#10;iQIzBAABCAAdFiEEnjBEBxt1jry35FZzcA5PObwFNksFAmO2D+QACgkQcA5PObwF&#10;Nkty6w/6A2ZeB18CSP7dVTSb0DFOmzA0Q3W/vhoAWm32unUzBDSUA7gloXp6KdqJ&#10;xDKn/28AYfGuFGcZBSN/FSVDWyXQcJD3/Idix4xjRPJ45HibzXej4Pfuomr3Fdi3&#10;TMeedRWfVJeXX1nTht5Wbx7FCU3TvG2f7jJcJnZc0x3CU8KEKrFNP/QAmcEvo2vZ&#10;6kuLSKBQ5093+UCNpu1YaGZA3nJm10j3AZ3pDzyB0bvb2tZ+ATAoNZ37NsgP98wm&#10;5f1l5mgwhG73yLjk1KLlujjUKvY552gRaCju3ceNKkfGtCSyLiZDjf2qoTn5l5mv&#10;IJJqvfHZVi3iUQa/8Jf/E64A1W+2wMKyLBHnOq4gKPrw2b3dLp+AXi+Ub3178q+a&#10;fwyULsqcny5RL6QpjynyoJJlB9q8PZmcajIY/rrvHi4peO3psm+wgqUEeaJGmf1Y&#10;BkIyIJ8R3xoBK1Hv7xVvl9wmKa12znC2CKHPda5jjGMMc3z6zQSngZhl73PQ/4P6&#10;KVjvhQEpJq+w0Zk5czZ9iJHXCqtjyCamGDvg5cLQt/Kw6Pkj1Pjgi01XQ3mJqD3B&#10;jormwEBrtsctSluJ02WGznbnZbE8r70UF/+T76qYHW303LUoDjTYDTyllxISZ01T&#10;+nYTmDArweB3Bb/whbkfMufMjHnH0oR+Pu8WAdEsAes0eT+tSEQ=&#10;=bbNj&#10;-----END PGP SIGNATURE-----'/>
        <property name='pgp.publicKeys' value='-----BEGIN PGP MESSAGE-----&#10;&#10;uQINBFhaXPsBEAC3bR7f5euHbpIDDTuFYHPI0+S5X0DhuqcGBUL2HSFhWMwIlfsA&#10;aO+pt7GyfXLUkTmzugwmwO+sOW2QmwEZQcK2z3BrcjytZophZ9AUajbAjnadSH6U&#10;XCMmfExVVnaYSfl/+Uub42szQE/r3gCRIz6M6clVVAjpFv4G/mumfQUV/XzLoUEY&#10;XTgwTokFJ97R+hDbHvBEBrUT8M6zHP5DhN3EBug3qb6wZVOa/+HEX3M+7k4jVT/p&#10;pNumw0acg0DDoSNQ13VsRV6sV0XE4zr3Zfs84f8xCgXpEMs4U6DZGqs3iJVVtbRf&#10;0oL0fgcxNgRrmbCrBfbXYfrS4u+fJ0vB+Wrflv9eNA3i6TtVL6uYpZy9uO2B1olK&#10;VzfEhsgB3QrULB4jVHZjIXGe4ILn45ndMtAeY4M91wyobgG99Xl+1vPHrxV0+2zR&#10;P66J3puyxiKE2B7gd7hib54CB3lYyrG1S+K1kZGCI1IFKCnqmTJXY0tKoLAASS3v&#10;tDcknXenzR5RVSpWTDuxtusekfL0Bw8pCBoz9L4Hex8Q1j//D5CZlqcg1NKFfmBZ&#10;7ta9PTuJcpOsz/LaPG/0VHYt/QAv5o4eeZESl7iZyM4/0NFh2s/rq0R8Z9yVSSkI&#10;vvO8d8XGZ65NTm3T4NFuEihn+AEm+zg4KiGdYBEZvs8QQoW9e1+MMN8xnwARAQAB&#10;iQREBBgBCAAPAhsCBQJhuzR9BQkSxtkCAinBXSAEGQEIAAYFAlhaXPsACgkQcA5P&#10;ObwFNkunSw//SRR1tGS1pDj2jonLpR0wPilCphS6ANv895yvlg6rHG4nKi4hQ0Jz&#10;ZxhGCwkgxEkRaKiyLfEiTihETkF161AqLPhyvE8LuQ1AG+A+tUnR8/T3gKE8t/m2&#10;/UtScZwN1QEQVc/uG7MTrbZ2ngXfH65k3fzhjy95AnJHAswu2vic1hzDi77HlQpN&#10;0O3adJuU/jfdu1RxNE0MRt8MFEjsTFwSBVm6lDxgcZV+qjRLGQznTyLF5/AyCI7Z&#10;4z9xHZPKFq1eHzqevifNiqfb8KX22sHKOSdnVBzBq/UxbT5jIbNSRhD91FjtZD7Z&#10;6wi3POsB/9RWZBldCov4ZEajmxFzxpx4RAqYOSIkEor9ZtRGbZuWvTie4vFIur7T&#10;f543mE6nxKcggExNp4MTyOd1scMc9oyczH561OTdHOCYEyoCwpG9N2Hb1/MDnWSi&#10;HKG451CvdrE5FHcPZKjp/nHUcRw/WQC3bgj6ScAay64EKC5S9tW+Wp85Oyyvj+M7&#10;lBzOxp19nESpfC++fzBAQPMxtD8EvrZTxqFSJxMOH9bhzB8+MFt08tmYb5SwoYi4&#10;C8JJ+wZgNetJKK+j07fvyMUChH/SbkCVszMiiSEjHA2Kk0LMVYKS/OLJU7i7tZXV&#10;aJ078QEeTDy5hSzsutd+orlFkR9+mgr1HUh0UgYlofTfEi7bLDeSr0cJELbTq5vM&#10;ZBKCicIP/irazYBVKw0SluhHtjzRcs5WIdH5bVPsEE87+iUc4daONWdVIhLdokxt&#10;OWlrEmZFLKqq9Z8fzvlf5LAQMOBkMAkl0z2ej4KG7zrjWyqDgysEI2WBlqTAFSeL&#10;+89Kc9BzJE9heYW8EfpXbNfOnKnAYWsbhcomSxVQ/jBIuyLBg/0gYKpBNx8HC6v9&#10;xNH0Ja+wM/7w3JC1aIwMYJn1yF2ykUYS+BoTCU7TA8r43pHg4I4Fz+Y2P5RLk+RJ&#10;I4kJezDNiJOpIcr/nKTPxMGUzMtWlGyAJ7LkyOZCtQXhtXwaT8grjtHzlwlGrpgD&#10;Rtf7wWjzEWeaQSegTFM9Mid+09kCp0PkJvveg8wJCuoVboNOto0O5rQsUczjXxiW&#10;kXYlHGeQL4rWc1zP7F1n4DEwDbVZC7jOn/80l3x4LcKuhc86gP4L5HKbdjn5GcQ0&#10;3RVLl1WVTQCdpr0+am28hl9XpyHdlWwSEmqqoUnjGv5B8RClocBRS4ECPPZCVSBl&#10;yK8eDgRww9Fu1EFq4xkq5fGj4YUOAIm756iW41NQ3VnPYbom/J27iFFN8+h92CSb&#10;KAqhmRwQh+GGo0eGCXmPHyQ/KCHTvnTZCFBUvabm3rVNFaDO+RvmwPwNCRz0DYzG&#10;paeMOGo4nMMGbzdhgfJ/X5Ed1/Mqz8egHhGIO94ebKEN5ZtJjAOK&#10;=xtSl&#10;-----END PGP MESSAGE-----'/>
      </properties>
    </artifact>

The underlying p2 infrastructure used by Tycho (a Tycho version that uses at least the 2022-03 release before which the PGP support was broken) and for installing or mirroring will transfer these properties from the source to the target. So yes, I do expect PGP signatures and keys to "follow" the artifact and that definitely generally works, e.g., the SimRel aggregator does nothing special to transfer the PGP key and signature properties, relying on p2 to do that....

[merks]

And yes, the goal here is to provide something such that downstream consumers can avoid needing to do PGP signing and such that it's easy to provide downstream consumers with additional dependencies upon request, with an overall goal to be easier/cheaper to maintain than is currently the case for Orbit.

[cdietrich]

is there an easy way to check if a p2 repo has valid pgp signature for a certain artifact?

[merks]

Open the artifacts.jar/artifacts.xml.xz and the artifacts.xml inside that. Find the <artifact classifier='osgi.bundle' id='*' version='*'> and check to see that it has pgp.signatures and pgp.publicKeys properties. If those properties are present, it's safe to assume they came from the originating p2 repository and are valid.

[LorenzoBettini]

@cdietrich, what I'm willing to try to do is, as I proposed in the past, switch entirely to Maven/Tycho. In my experience, that should simplify lots of our building parts, not to mention that the p2 repositories generated will not have the problems we have experienced lately. Also, my projects (both the official Eclipse projects and other projects I maintain) have an involved build structure, which I can easily manage with Maven/Tycho. I'm willing to do that as long as there's interest from the Xtext committers.

[cdietrich]

@LorenzoBettini i am not sticking to any technology here as long as we are still compatibly consumable from pure maven and gradle builds and thus wont break these clients.

@szarnekow i assume you would be fine with pure tycho builds as well.

[merks]

I can also help with pure Tycho builds having setup and maintained a large number of them over the past years.

[LorenzoBettini]

@LorenzoBettini i am not sticking to any technology here as long as we are still compatibly consumable from pure maven and gradle builds and thus wont break these clients.

Yes, my artifacts are available from p2 repositories and Maven central. (everything signed, of course). I typically deploy to Maven central artifacts without the build qualifier (e.g., 2.12.1), while on p2 repositories, I leave the full build qualifier (e.g., 2.12.1.v202212...blah blah). But that's just my design choice.

[cdietrich]

regardning repackaging looks good with tycho 2.7.5 and tycho 3.0.1: https://github.com/cdietrich/repackage-pgp-signed-stuff-experiments

[szarnekow]

@szarnekow i assume you would be fine with pure tycho builds as well.

Absolutely.

I'd go even a step further:: If things get simpler by merging everything into a single repo, I'm willing to do that, too.

[LorenzoBettini]

I'd go even a step further:: If things get simpler by merging everything into a single repo, I'm willing to do that, too.

I didn't dare to say that, but I'd love to try that one as well :)

[merks]

FYI, I merged the EMF and XSD repositories, preserving the history. It wasn't so hard...

[cdietrich]

the question is, how to stack the stuff on each other,
first monorepo, then adapt existing builds
then tycho build?

[LorenzoBettini]

Yes, that could be a possible workflow.
Alternatively, I'd start porting to Maven/Tycho the "core" and then go on.
However, having everything in the same repo will likely make the port to Maven/Tycho easier due to some inter-dependencies.
On the other hand, the merge to a single repo might be more disruptive...
just a few random thoughts.

@merks, how have you implemented merging?
My idea was something like that:

• start from repo A
• add repo B as a remote (and I have two separate streams in the Git repository without intersections)
• merge repo B into A (and I now have a merging point of the two streams)
  Is this what you've done?
  Or are there any automatic tools?

[merks]

I needed some help at the end from the webmaster:

https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/2444

As mentioned there, I followed these simple instructions which sounds like what you are describing:

https://jeffkreeftmeijer.com/git-combine/

It was way simpler than I though it would be...

Also this helped simplify the build so I am able to use a multi-branch pipeline:

https://ci.eclipse.org/emf/job/build/

I also got rid of EMF's specialized site promoter/generator and reused the JustJ-provided one instead.

[cdietrich]

i guess we can start experimenting with core, then we might see what makes problems and what works

[LorenzoBettini]

i guess we can start experimenting with core, then we might see what makes problems and what works

do you mean to experiment with Maven/Tycho with "core" or to try to merge "core" with another Git repository?

[cdietrich]

experiment with tycho

[LorenzoBettini]

Note that Oomph now does support including a normal *.target into a Targlet container and that target can contain Maven dependencies. This work was funding by https://www.diamond.ac.uk/

@merks this means however that Oomph cannot yet support Maven coordinates (as we do in the .target files) directly, is that right?

The approach I described in the forum will allow Maven dependencies in a centrally managed *.target file (along with distributed managed additional *.target files from SimRel projects) to be used to generate a normal p2 repository that can be consumed in the normal way exactly as you are currently consuming from Orbit. Mind you, the artifacts are PGP-signed, not jar-signed.

Could you please point me to the forum post you're mentioning?

In general, would you suggest switching to target files in Oomph as well?

[merks]

Sorry I thought I had added a link to the Bugzilla that has a PDF attachment with documentation

https://bugs.eclipse.org/bugs/show_bug.cgi?id=580054
https://bugs.eclipse.org/bugs/attachment.cgi?id=288551

WindowBuilder uses it:

https://github.com/eclipse/windowbuilder/blob/master/setups/WindowBuilder.setup

This target

https://github.com/eclipse/windowbuilder/blob/master/target-platform/wb.target

Includes this target:

https://github.com/eclipse/windowbuilder/blob/master/target-platform/mvn/wb-mvn.target

And that one is also used by Oomph's targlets.

I had to add this task

https://github.com/eclipse/windowbuilder/blob/06ff485102ce83f83082dc2d9ef0e6720286afc5/setups/WindowBuilder.setup#L117-L121

So that the wb-mvn.target is available in the workspace for use here:

https://github.com/eclipse/windowbuilder/blob/06ff485102ce83f83082dc2d9ef0e6720286afc5/setups/WindowBuilder.setup#L141

I'm hoping get what I described on cross-projects working over the weekend, in which case I could just add what you need so that it's in a regular p2 repository. Is there something specifically that you needed that's not also used directly by the platform?

[LorenzoBettini]

@merks thanks for the answer and sorry for my delay. While porting Xtext projects to Maven/Tycho there were a few projects that needed dependencies not available from Orbit and as a first attempt, I used Maven dependencies in the .target file. Later, I realized that for those specific (test) projects, I could go completely Maven (no PDE), so dependencies are taken directly from Maven Central. So for the moment the .target only contains p2 units.

[merks]

This this available, I think all is prepared:

https://download.eclipse.org/oomph/simrel-orbit/

I'm not sure what's left to do for this issue....