Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TCK challenge for adding support for running Security 3.0 signature TCK tests on Java 21 #297

Open
scottmarlow opened this issue Dec 1, 2023 · 15 comments
Labels
accepted Accepted certification request challenge TCK challenge was appealed

Comments

@scottmarlow
Copy link

When running the https://www.eclipse.org/downloads/download.php?file=/jakartaee/security/3.0/jakarta-security-tck-3.0.1.zip tests on Java 21, the following signature test failures occur due to Java 21 added a few new JDK methods:

12-01-2023 11:32:57:********** Status Report 'jakarta.security.enterprise.identitystore.openid' **********
12-01-2023 11:32:57:SignatureTest report
Base version: 3.0.0
Tested version: 3.0.0
Check mode: src [throws normalized]
Constant checking: on

Added Superclasses or Superinterfaces

jakarta.security.enterprise.identitystore.openid.Scope: interface java.util.SequencedCollection<java.lang.String>
jakarta.security.enterprise.identitystore.openid.Scope: interface java.util.SequencedSet<java.lang.String>

Added Methods

jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.getFirst()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.getLast()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.removeFirst()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.removeLast()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.util.SequencedSet<java.lang.String> java.util.LinkedHashSet.reversed()
jakarta.security.enterprise.identitystore.openid.Scope: method public static <%0 extends java.lang.Object> java.util.HashSet<{%%0}> java.util.HashSet.newHashSet(int)
jakarta.security.enterprise.identitystore.openid.Scope: method public static <%0 extends java.lang.Object> java.util.LinkedHashSet<{%%0}> java.util.LinkedHashSet.newLinkedHashSet(int)
jakarta.security.enterprise.identitystore.openid.Scope: method public void java.util.LinkedHashSet.addFirst(java.lang.String)
jakarta.security.enterprise.identitystore.openid.Scope: method public void java.util.LinkedHashSet.addLast(java.lang.String)

12-01-2023 11:32:57:********** Package 'jakarta.security.enterprise.identitystore.openid' - FAILED (STATIC MODE) **********

12-01-2023 11:32:58:********** Status Report 'jakarta.security.enterprise.identitystore.openid' **********

12-01-2023 11:32:58:SignatureTest report
Base version: 3.0.0
Tested version: 3.0.0
Check mode: src [throws normalized]
Constant checking: on

Added Superclasses or Superinterfaces

jakarta.security.enterprise.identitystore.openid.Scope: interface java.util.SequencedCollection<java.lang.String>
jakarta.security.enterprise.identitystore.openid.Scope: interface java.util.SequencedSet<java.lang.String>

Added Methods

jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.getFirst()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.getLast()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.removeFirst()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.removeLast()
jakarta.security.enterprise.identitystore.openid.Scope: method public java.util.SequencedSet<java.lang.String> java.util.LinkedHashSet.reversed()
jakarta.security.enterprise.identitystore.openid.Scope: method public static <%0 extends java.lang.Object> java.util.HashSet<{%%0}> java.util.HashSet.newHashSet(int)
jakarta.security.enterprise.identitystore.openid.Scope: method public static <%0 extends java.lang.Object> java.util.LinkedHashSet<{%%0}> java.util.LinkedHashSet.newLinkedHashSet(int)
jakarta.security.enterprise.identitystore.openid.Scope: method public void java.util.LinkedHashSet.addFirst(java.lang.String)
jakarta.security.enterprise.identitystore.openid.Scope: method public void java.util.LinkedHashSet.addLast(java.lang.String)

12-01-2023 11:32:58:SigTestEE.signatureTest() failed!, diffs found
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.252 s <<< FAILURE! - in ee.jakarta.tck.security.signaturetest.SecurityAPISigTestIT
[ERROR] ee.jakarta.tck.security.signaturetest.SecurityAPISigTestIT.signatureTest Time elapsed: 6.074 s <<< ERROR!
ee.jakarta.tck.security.signaturetest.SigTestEE$Fault: SigTestEE.signatureTest() failed!, diffs found
at [email protected]/ee.jakarta.tck.security.signaturetest.SecurityAPISigTestIT.signatureTest(SecurityAPISigTestIT.java:355)

There are two possible fixes, the easiest one line change would be to exclude the specific updated JDK classes on the command line when invoking the sigtest tool. I'll look into that next year.

The better change would be to update https://github.com/jtulach/netbeans-apitest to be able to avoid checking any SPEC API classes that are not in the jakarta namespace which I think would avoid validation of any JDK classes and instead focus only on checking jakarta.* classes. I'd like to do this change for Jakarta EE 11+.

@scottmarlow
Copy link
Author

Since all of the Signature test failures also happen in the Platform TCK signature tests we can probably address that first and likely make the same one line change in other places as well. I think we should turn these issues into TCK challenges so could one of the contributors please add the challenge label. Thanks!

@scottmarlow scottmarlow changed the title Support for running signature TCK tests on Java 21 TCK challenge for adding support for running signature TCK tests on Java 21 Dec 5, 2023
@scottmarlow scottmarlow changed the title TCK challenge for adding support for running signature TCK tests on Java 21 TCK challenge for adding support for running Security 3.0 signature TCK tests on Java 21 Dec 5, 2023
@scottmarlow
Copy link
Author

I can confirm that the jakartaee/platform-tck#1206 change solved ^ running the (Jakarta EE 10) Platform TCK Signature tests on Java 21 and the same change should help the Security TCK.

@scottmarlow
Copy link
Author

Since all of the Signature test failures also happen in the Platform TCK signature tests we can probably address that first and likely make the same one line change in other places as well. I think we should turn these issues into TCK challenges so could one of the contributors please add the challenge label. Thanks!

^ is done and the updated Platform TCK 10.0.3 TCK is released as mentioned on https://jakarta.ee/specifications/platform/10

@scottmarlow
Copy link
Author

Could one of the Security project committers please add the challenge label and initiate process to accept or reject this TCK challenge.

@arjantijms arjantijms added challenge TCK challenge was appealed accepted Accepted certification request labels Apr 9, 2024
@arjantijms
Copy link
Contributor

@scottmarlow sorry for the delay. Of course this will be done for the final release of the TCK. I just did the same for the Jakarta Authorization TCK.

See: https://github.com/jakartaee/authorization/tree/master/tck/authorization-signaturetest

@scottmarlow
Copy link
Author

@scottmarlow sorry for the delay. Of course this will be done for the final release of the TCK. I just did the same for the Jakarta Authorization TCK.

See: https://github.com/jakartaee/authorization/tree/master/tck/authorization-signaturetest

@arjantijms to be clear, I think that we need a https://www.eclipse.org/downloads/download.php?file=/jakartaee/security/3.0/jakarta-security-tck-3.0.2.zip release that includes the #305 fix as well.

@arjantijms
Copy link
Contributor

@scottmarlow okay, that should be doable

@arjantijms
Copy link
Contributor

@scottmarlow
Copy link
Author

I've staged 3.0.2 here, but not checked it yet:

https://www.eclipse.org/downloads/download.php?file=/security/jakartaee10/staged/eftl/jakarta-security-tck-3.0.2.zip

Thanks, I'm working on running it locally with latest WildFly on Java 17 and will then run again on Java 21.

@arjantijms
Copy link
Contributor

The JDK 21 run against GlassFish 7.0.15 succeeded: https://ci.eclipse.org/jakartaee-tck/view/EFTL-Certification-Jobs-10/job/10/job/jakarta-security-tck-glassfish/141/

@scottmarlow
Copy link
Author

I'm hitting environmental problems locally so probably a good idea to promote it as I did at least see the signature tests pass locally with JDK21:

All package signatures passed.
Passed packages listed below:
jakarta.security.enterprise(static mode)
jakarta.security.enterprise(reflection mode)
jakarta.security.enterprise.authentication.mechanism.http(static mode)
jakarta.security.enterprise.authentication.mechanism.http(reflection mode)
jakarta.security.enterprise.authentication.mechanism.http.openid(static mode)
jakarta.security.enterprise.authentication.mechanism.http.openid(reflection mode)
jakarta.security.enterprise.credential(static mode)
jakarta.security.enterprise.credential(reflection mode)
jakarta.security.enterprise.identitystore(static mode)
jakarta.security.enterprise.identitystore(reflection mode)
jakarta.security.enterprise.identitystore.openid(static mode)
jakarta.security.enterprise.identitystore.openid(reflection mode)

@arjantijms
Copy link
Contributor

JDK 11 and JDK 17 runs passed too. I'll ask for the promotion.

@scottmarlow
Copy link
Author

@arjantijms any idea why Apache Tomcat might be used for some tests by default? I previously tested with the 3.0.0 TCK but not 3.0.1 (in case that helps). https://gist.github.com/scottmarlow/50802d6a60ca5228fa92d207b4d44a78 shows some evidence of org.apache.catalina.loader.WebappClassLoaderBase being used in my 3.0.2 TCK testing which sounds config related.

I also saw the first 3.0.2 TCK test failure with error ee.jakarta.tck.security.test.OpenId2DefaultIT which seems to be caused by Failed to start service org.wildfly.undertow.listener.https: org.jboss.msc.service.StartException in service org.wildfly.undertow.listener.https: Address already in use /127.0.0.1:8443 which is what I saw locally and thought I configured something wrong (so tried running on Jenkins but get the same failure there).

@scottmarlow
Copy link
Author

Perhaps ^ problem with Tomcat automatically starting for some tests is related to the #284 change for 3.0.1.

@arjantijms
Copy link
Contributor

any idea why Apache Tomcat might be used for some tests by default?

Yes, two tests use an OpenID Connect Server to test against, and this just happens to be a Tomcat application. It's similar to how e.g. the Jakarta Mail TCK starts up a Mail Server to test against.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted Accepted certification request challenge TCK challenge was appealed
Projects
None yet
Development

No branches or pull requests

2 participants