Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[release]: JDBC driver shadow jar v.1.3.0.0 #3614

Closed
Yury-Fridlyand opened this issue Jun 9, 2023 · 29 comments
Closed

[release]: JDBC driver shadow jar v.1.3.0.0 #3614

Yury-Fridlyand opened this issue Jun 9, 2023 · 29 comments
Labels

Comments

@Yury-Fridlyand
Copy link
Contributor

Did you read the on-boarding document

What is the name of your component?

JDBC driver

What is the link to your GitHub repo?

https://github.com/opensearch-project/sql-jdbc

Targeted release date

Where should we publish this component?

artifacts: https://artifacts.opensearch.org/
download page: https://opensearch.org/downloads.html#drivers

What type of artifact(s) will be generated for this component?

Shadow jar

Have you completed the required reviews including security reviews, UX reviews?

Have you on-boarded automated security scanning for the GitHub repo associated with this component?

Additional context

sql-jdbc.zip
JDBC driver v.1.3.0.0 was released on maven, but we should also publish shadow jar of this version on the main website. See attached artifact for this version.
This jar should be signed prior to publish.

@Yury-Fridlyand Yury-Fridlyand added release untriaged Issues that have not yet been triaged labels Jun 9, 2023
@Yury-Fridlyand
Copy link
Contributor Author

This blocks Tableau connector release.
@gaiksaya, could you have a look please?

@gaiksaya
Copy link
Member

Hi @Yury-Fridlyand ,

Few questions:

  1. The zip only needs to be published on website or also on maven?
  2. In order to sign and publish the artifact anywhere we would automate the artifact creation, signing and uploading. Hence, wondering if this can be included in 1-click release process of sql-jdbc release a well?
  3. Is this one time process or there are further releases planned for this?

@gaiksaya gaiksaya removed the untriaged Issues that have not yet been triaged label Jun 12, 2023
@Yury-Fridlyand
Copy link
Contributor Author

Yury-Fridlyand commented Jun 12, 2023

  1. Shadow jar goes to website only. Note: a jar should be published, not a zip. GH doesn't allow to upload jar files. *
  2. Yes, it would be awesome.
  3. No, we should do this with every release of JDBC.

Non shadow jar is already on maven and it is ok, we should publish both of them.

@gaiksaya
Copy link
Member

gaiksaya commented Jun 12, 2023

Thanks!
Only 1.13.0.0 jar needs to be published to website but going forward both jars can be published to maven as well as website if I am not wrong?

The link on the website jar is still pointing to opensearch-sql-jdbc-1.1.0.1.jar which is very old. Is someone following up with website team on this?

We would need to make changes to add the shadow publication jar to this tarball https://github.com/opensearch-project/sql-jdbc/blob/main/.github/workflows/release-drafter.yml#L24
Signing and publishing to maven is automatically taken care of by jenkins file

However, we need to add publishToArtifactsProdBucket to jenkins file that will take care of uploading the artifacts to S3 bucket/website automatically.

Tagging @rishabh6788 who would be taking care of this.

Thanks!

@Yury-Fridlyand
Copy link
Contributor Author

going forward both jars can be published to maven as well

shadow jar goes to website (to artifacts actually)
non shadow jar goes to maven (as it works now)

I think I can trigger updating link on the website on my own, once it is uploaded to artifacts.

Thanks @gaiksaya and @rishabh6788!

@rishabh6788
Copy link
Collaborator

@Yury-Fridlyand Could you please share the command to generate shadow jars and also does it generate the artifacts in the repository directory or is it on a different path?

@Yury-Fridlyand
Copy link
Contributor Author

./gradlew shadowJar generates opensearch-sql-jdbc-1.3.0.0.jar in build/libs.

@rishabh6788
Copy link
Collaborator

I tried the above mentioned command and it generated opensearch-sql-jdbc-1.4.0.0.jar jar in build/libs dir.
I then ran ./gradlew publishPublishMavenPublicationToLocalRepoRepository and it also generated the new jar with same name and overwrote the one generated using shadowJar command.

  1. Can we rename the shadow jar to something like opensearch-sql-jdbc-1.4.0.0-shadow.jar to avoid the confusion of having two jars at different locations with the same name?
  2. Is the behavior wrt to publishPublishMavenPublicationToLocalRepoRepository overwriting the jar created using shadowJar expected?

@Yury-Fridlyand Yury-Fridlyand changed the title [release]: JDBC driver shadow jar [release]: JDBC driver shadow jar v.1.3.0.0 Jun 14, 2023
@Yury-Fridlyand
Copy link
Contributor Author

@rishabh6788 you probably checked out main, you need to check out 1.3.0.0 release.
Answering your questions:

  1. I think yes
  2. Yes, both commands create an artifact

There is no need to do publishPublishMavenPublicationToLocalRepoRepository for getting a shadow jar.

@Yury-Fridlyand
Copy link
Contributor Author

This is required for #3631

@gaiksaya
Copy link
Member

There is no need to do publishPublishMavenPublicationToLocalRepoRepository for getting a shadow jar.

Hey @Yury-Fridlyand 1.13.0.0 apart we are trying to include this in 1-click release process which sql-jdbc is already onboarded too. So the idea is to include it in the same workflow and just bundle both (shadow, non-shadow jars) into artifacts.tar.gz to be supplied to jenkins workflow. Here to be precise: https://github.com/opensearch-project/sql-jdbc/blob/main/.github/workflows/release-drafter.yml#L24
So from next release, both will be released simultaneously.

@rishabh6788
Copy link
Collaborator

@Yury-Fridlyand If I remember correctly tableau needs the artifacts to be signed using JarSigner with timestamp if you are planning to upload to Tableau marketplace.
What we are using is a PGP signer to upload the signed jar to artifacts bucket.
Can you please confirm the signing requirement?

@Yury-Fridlyand
Copy link
Contributor Author

Oh, that is very good point. Right, Tableau require JarSigner's signature.
Is is possible to apply two signatures and have them valid on a single jar? If not, we have to publish 2 shadow jars - signed by JarSigner and by PGP signer. Both should be available for public to download.

@rishabh6788
Copy link
Collaborator

I believe even the jar signed by JarSigner can be used by public, both forms signing represent that it comes from a valid source and there is no change in the functionality.

@rishabh6788
Copy link
Collaborator

Since the already uploaded Jar on opensearch.org has been signed using PGP, in order to not break the signing method mentioned on the website we will have to sign it using the existing PGP method and also upload a JarSigner signed copy as well for Tableau.
@gaiksaya @bbarani Let me know your thoughts.

@Yury-Fridlyand
Copy link
Contributor Author

Great
I also confirm that jar file name could be renamed for Tableau submission, if is needed.

As I see total JDBC release process it would like this:

./gradlew publishPublishMavenPublicationToLocalRepoRepository
# sign
...
# upload to maven
...

./gradlew shadowJar
# PGP sign
...
# Upload to artifacts
...

./gradlew shadowJar
# rename
mv opensearch-sql-jdbc-$VERSION.jar opensearch-sql-jdbc-$VERSION-shadow-jarsigner.jar
# JarSigner sign
...
# Upload to artifacts
...

# Update downloads page
...

@rishabh6788
Copy link
Collaborator

rishabh6788 commented Jun 14, 2023

opensearch-sql-jdbc-1.3.0.0-shadow.jar.zip
@Yury-Fridlyand We decided to go with one shadow jar that can be used by public and tableau, signed using Jarsigner. We need to update the documentation accordingly on the Downloads page.
I have attached the signed jar, can you please download and verify from your end and make sure it is working as expected before we upload it to artifacts bucket.

@Yury-Fridlyand
Copy link
Contributor Author

Yes, I confirm that it works, but the signature is incorrect.
It is signed by Amazon Web Services, Inc., but should be OpenSearch Project.

@bbarani
Copy link
Member

bbarani commented Jun 14, 2023

@Yury-Fridlyand Currently we use the AWS signing pipeline as we are yet to implement OpenSource solution for our signing pipeline. It will be signed by Amazon Web Services, Inc. until that happens.

@rishabh6788
Copy link
Collaborator

@Yury-Fridlyand Signed shadow jar has been uploaded to artifacts bucket.

@Yury-Fridlyand
Copy link
Contributor Author

Awesome, I can download it: https://artifacts.opensearch.org/opensearch-clients/jdbc/opensearch-sql-jdbc-1.3.0.0-shadow.jar
Can I proceed with updating website link or we have to publish signature verification instructions for jarsigner first?

@rishabh6788
Copy link
Collaborator

For now let us remove the existing signature verification steps mentioned and instead state that the artifact is signed using JarSigner.
@bbarani

@acarbonetto
Copy link

@rishabh6788 the jarsigned artefact is verified and ready to put on the website. Would you like us to raise a PR to update the website?

@gaiksaya
Copy link
Member

For now let us remove the existing signature verification steps mentioned and instead state that the artifact is signed using JarSigner. @bbarani

Maybe add steps to verify that signature? @Yury-Fridlyand We can start with creating the PR to update the website.
Thanks!

@Yury-Fridlyand
Copy link
Contributor Author

Awesome!

@rishabh6788
Copy link
Collaborator

@rishabh6788 the jarsigned artefact is verified and ready to put on the website. Would you like us to raise a PR to update the website?

Yes, please go ahead. @acarbonetto
Do you need the command to verify using JarSigner that we can add for users to verify?

@Yury-Fridlyand
Copy link
Contributor Author

Yury-Fridlyand commented Jun 19, 2023

I used jarsigner -verify <jar> and jarsigner -verify -verbose <jar> to see the signer information.
Issue to publish JDBC on website for tracking: opensearch-project/project-website#1721

@gaiksaya
Copy link
Member

Hi @Yury-Fridlyand ,

Can you create a new issue to onboard the jdbc shadow jar publication automation to 1-click release process?
I can close this issue as the artifacts are release and related website PR is merged as well.

Thanks!

@Yury-Fridlyand
Copy link
Contributor Author

Created opensearch-project/sql-jdbc#98 for this. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants