gcs-connector:3.0.0 failing due to certificate when accessing to GCS from Github runner with WIF configuration #1106
Comments
|
We are seeing the same issue. The connector works fine locally, so maybe this is something specific to the GitHub Actions environment? |
|
Tried running our bootstrapping process and tests on a clean Ubuntu image, it also worked fine there. There must be something with the GitHub Actions image that causes this certificate issue. |
|
Hi @GergelyKalmar, Sorry, missed your comments. |
|
Yes, I don't think this feature is working properly. Sadly this library does not seem to be maintained much, at least by looking at the open issues and the lack of responses. |
|
@davidrabinowitz @cnauroth We would really appreciate if this library could get a little more attention. In particular, while we see maintainers dabble around in the code, our issues and comments are not addressed for months, and features that were released do not seem to be actually working. |
|
TL;DR: The HadoopCredentialsConfiguration provides a NetHttpTransport that uses a trust store that contains roots that work for Google APIs, but does not include the DigiCert roots needed for some other providers (e.g., github). While this thread is about WORKLOAD_IDENTITY_FEDERATION_CREDENTIAL_CONFIG_FILE, the google auth github action sets up ADC as well and that transport would also have issues. For this issue in particular, https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/master/util-hadoop/src/main/java/com/google/cloud/hadoop/util/HadoopCredentialsConfiguration.java#L255 should be changed to not include a net transport or the SSL context / trust store should also include CA certificates / certificates defined in javax.net.ssl.trustStore. My notes:
At this point in time, I see the following digicert itnermediate:
JDK8: Some older java8's will have this in sha1, you can grep for the sha1 (DF:3C:24:F9:BF:D6:66:76:1B:26:80:73:FE:06:D1:CC:8D:4F:82:A4) instead of the sha256 above instead if that's the case.
import com.google.api.client.googleapis.GoogleUtils;
import com.google.api.client.http.GenericUrl;
import com.google.api.client.http.HttpResponseException;
import com.google.api.client.http.javanet.NetHttpTransport;
import java.security.cert.X509Certificate;
import org.apache.commons.codec.digest.DigestUtils;
public class Test {
public static void main(String[] args) throws Exception {
var store = com.google.api.client.googleapis.GoogleUtils.getCertificateTrustStore();
var aliases = store.aliases();
while (aliases.hasMoreElements()) {
var alias = aliases.nextElement();
System.out.println("Alias: " + alias);
var entry = store.getCertificate(alias);
System.out.println("Format: " + entry.getPublicKey().getFormat());
if (entry instanceof X509Certificate) {
X509Certificate x509 = (X509Certificate) entry;
System.out.println("Issuer: " + x509.getIssuerX500Principal().getName());
System.out.println("Subject: " + x509.getSubjectX500Principal().getName());
System.out.println("Fingerprint sha256: " + DigestUtils.sha256Hex(x509.getEncoded()));
System.out.println("Fingerprint sha1: " + DigestUtils.sha1Hex(x509.getEncoded()));
}
System.out.println();
}
System.out.println("Connecting using default NetHttpTransport");
connectWithDefaults();
System.out.println(
"Connecting using NetHttpTransport with GoogleUtils.getCertificateTrustStore.");
connectWithGoogleStore();
}
public static void connectWithDefaults() throws Exception {
NetHttpTransport transport = new NetHttpTransport();
var req = transport.createRequestFactory()
.buildGetRequest(new GenericUrl("https://pipelinesghubeus6.actions.githubusercontent.com"));
try {
var resp = req.execute();
System.out.println("Defaults response code (expect an error): " + resp.getStatusCode());
} catch (HttpResponseException resp) {
System.out.println("Defaults response code (expect an error): " + resp.getStatusCode());
}
}
public static void connectWithGoogleStore() throws Exception {
NetHttpTransport transport = new NetHttpTransport.Builder().trustCertificates(
GoogleUtils.getCertificateTrustStore()).build();
var req = transport.createRequestFactory()
.buildGetRequest(new GenericUrl("https://pipelinesghubeus6.actions.githubusercontent.com"));
var resp = req.execute();
System.out.println("Google trust store code: " + resp.getStatusCode());
}
}
The final output of the above class: Overall, some external ID providers probably work and some will fail. |
|
Thank you @AngusDavis for your thorough investigation! Could somebody fix the SSL context / trust store accordingly? WIF should definitely work with major CI/CD environments like GitHub Actions. |
Hello,
We face below issue when application code connects to GCS from Github runner with WIF configuration. It is failing due to certificate. WIF - workflow identity federation which uses external account. As per documentation nothing points to certificate.
Any idea on that?
Thank you.
Spark config - scala example code:
.set("google.cloud.auth.type", "WORKLOAD_IDENTITY_FEDERATION_CREDENTIAL_CONFIG_FILE") .set("google.cloud.auth.workload.identity.federation.credential.config.file", <runtime_credential_json_path>) .set("fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS")Workflow logs:
com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getObject(GoogleCloudStorageImpl.java:1986) at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getItemInfo(GoogleCloudStorageImpl.java:1882) at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystemImpl.getFileInfoInternal(GoogleCloudStorageFileSystemImpl.java:861) at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystemImpl.getFileInfo(GoogleCloudStorageFileSystemImpl.java:833) at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.getFileStatus(GoogleHadoopFileSystem.java:724) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1862) at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.exists(GoogleHadoopFileSystem.java:817) at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$4(DataSource.scala:784) at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$4$adapted(DataSource.scala:782) at org.apache.spark.util.ThreadUtils$.$anonfun$parmap$2(ThreadUtils.scala:372) ... Cause: java.io.IOException: Error getting subject token from metadata server: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.IdentityPoolCredentials.getSubjectTokenFromMetadataServer(IdentityPoolCredentials.java:242) at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.IdentityPoolCredentials.retrieveSubjectToken(IdentityPoolCredentials.java:188) at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.IdentityPoolCredentials.refreshAccessToken(IdentityPoolCredentials.java:169) at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:257) at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:254) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.OAuth2Credentials$RefreshTask.run(OAuth2Credentials.java:623) at com.google.cloud.hadoop.repackaged.gcs.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:571) at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:220) ... Cause: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:360) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:303) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:298) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) at java.base/sun.security.ssl.CertificateStatus$CertificateStatusConsumer.consume(CertificateStatus.java:295) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)The text was updated successfully, but these errors were encountered: