spring.cloud.gcp.credentials.encoded-key Not working

[rajivml]

HI,

Am trying to pass the credentials via the spring.cloud.gcp.credentials.encoded-key property,

and this is command am using to convert the key file to an base64 encoded value as suggested in this blog https://cloud.google.com/kms/base64-encoding

base64 key_file.json -w 0

key_file.json contents:
{
  "type": "service_account",
  "project_id": "xxxxx",
  "private_key_id": "xxxxxxxxxxxxxxxx",
  "private_key": "-----BEGIN PRIVATE KEY-----xxxxxxxxxxx\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "xxxx",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/spring-sleuth%xxxxx.iam.gserviceaccount.com"
}

And my Application is crashing always with the below error, any Idea what could be the reason for this issue, and do i need to encode the entire service account JSON file or the JSON with only private_key property ?

[com.google.api.gax.core.CredentialsProvider]: Factory method 'googleCredentials' threw exception; nested exception is java.lang.IllegalArgumentException: Illegal base64 character 7b at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:769) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:509) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1288) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1127) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:204) at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:235) at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:226) at

[rajivml]

If I mount the same key as a file through SPRING_CLOUD_GCP_CREDENTIALS_LOCATION it's working fine but when I set the same key through spring.cloud.gcp.credentials.encoded-key so that mounting isn't required and testing locally will also become easy

But it's not working and throwing the above exception posted

[meltsufin]

@rajivml As I understand, SPRING_CLOUD_GCP_CREDENTIALS_LOCATION is equivalent to spring.cloud.gcp.credentials.location. Have you tried using that instead of spring.cloud.gcp.credentials.encoded-key?

[rajivml]

When I am passing the key file through SPRING_CLOUD_GCP_CREDENTIALS_LOCATION and SPRING_CLOUD_GCP_TRACE_CREDENTIALS_LOCATION, it's working fine, but I need to create a volume and mount that volume to the container for it to work.

And if am testing the app locally I need to make sure that I have the key file on my local system and my SPRING_CLOUD_GCP_CREDENTIALS_LOCATION environment variable is point to that location.

That's why I am keen on using spring.cloud.gcp.credentials.encoded-key property where I converted the JSON key file into base64 encoded string but it was throwing the exception which I have mentioned above.

Did you get a chance to try it and replicate the issue ?

[meltsufin]

You need the full contents of the JSON file to be base-64 encoded.
This is the code that decodes it and creates the credentials:

GoogleCredentials.fromStream(
   new ByteArrayInputStream(Base64.getDecoder().decode(encodedKey)))
        .createScoped(scopes));

See: https://googleapis.github.io/google-auth-library-java/releases/latest/apidocs/com/google/auth/oauth2/GoogleCredentials.html#fromStream-java.io.InputStream-

[rajivml]

I have encoded the complete JSON file itself... will check what the below piece is returning and get back

Base64.getDecoder().decode(encodedKey)))
.createScoped(scopes)

[meltsufin]

@rajivml I just tried using base64 [file] -w 0 and setting the ecoded-key. It worked for me.

[meltsufin]

Closing, since we're unable to reproduce. Please let us know if you find out what was wrong.

[AbhinavAtul]

if using multiple profiles, in the same application.yml in particular, make sure the profile that you are running locally has the credentials configured

[fragaLY]

@meltsufin

Hello, seems like the issue is still there.

The example of my setup doesn't works correctly:
1st of all I've created a key for gcp storage and encoded it using (macOS):
base64 -i key.json -b -

After that I've passed the encoded key into profile:
Encoded key sample:
eAiLS0tLS1CRUdJT.....SQBSFIJ!N KF!LJA=

My application.yaml:

server:
  port: 8081
  undertow:
    threads:
      io: 2
      worker: 16
    max-http-post-size: -1B
    buffer-size: 16KB
    direct-buffers: on
  shutdown: graceful
  servlet:
    session:
      timeout: 60m
      cookie:
        http-only: true
    application-display-name: api
  error:
    whitelabel:
      enabled: false

spring:
  application:
    name: api
  main:
    lazy-initialization: true
    web-application-type: servlet
    banner-mode: off
  jackson:
    time-zone: UTC
    locale: en_US
  datasource:
    driverClassName: org.postgresql.Driver
    url: ${DB_URL}
    username: ${DB_USER}
    password: ${DB_PASSWORD}
    hikari:
      connection-timeout: 1000
      pool-name: "hikari-pool"
  flyway:
    enabled: true
    locations: classpath:db/migration
    url: ${DB_FLYWAY_URL}
    user: ${DB_USER}
    password: ${DB_PASSWORD}
    connect-retries: 3
    schemas: "api"
  data:
    jpa:
      repositories:
        bootstrap-mode: deferred
  jpa:
    open-in-view: false
    hibernate:
      ddl-auto: none
    generate-ddl: false
    database-platform: org.hibernate.dialect.PostgreSQL95Dialect
    properties:
      hibernate:
        default_schema: ${DB_SCHEMA}
    show-sql: true
  mvc:
    format:
      date: iso
      date-time: iso
  lifecycle:
    timeout-per-shutdown-phase: 30s
  autoconfigure:
    exclude: org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration
  servlet:
    multipart:
      enabled: true
      max-file-size: -1
      max-request-size: -1
      resolve-lazily: true
  cloud:
    gcp:
      storage:
        enabled: false

management:
  health:
    livenessstate:
      enabled: true
    readinessstate:
      enabled: true
  endpoint:
    health:
      enabled: true
      probes:
        enabled: true
      show-components: never
      show-details: never
      group:
        readiness:
          include: readinessState, db
    metrics.enabled: true
    prometheus.enabled: true
  endpoints.web.exposure.include: "*"
  metrics.export.prometheus.enabled: true

logging.level:
  ROOT: info
  by.vk.api: info
  org.springframework: info

springdoc:
  show-actuator: true
  api-docs:
    enabled: true
    groups:
      enabled: true

---
spring:
  config:
    activate:
      on-profile: prod
  datasource:
    url: ${DB_PROD_URL}
    username: ${DB_PROD_USER}
    password: ${DB_PROD_PASSWORD}
  flyway:
    enabled: true
    url: ${DB_PROD_FLYWAY_URL}
    user: ${DB_PROD_USER}
    password: ${DB_PROD_PASSWORD}
  jpa:
    show-sql: false
  cloud:
    gcp:
      storage:
        enabled: true
        project-id: ${GCP_PROJECT_ID}
        auto-create-files: true
        credentials:
          encoded-key: eAiLS0tLS1CRUdJT.....SQBSFIJ!N KF!LJA=

If I replace encoded-key with key.json location: key.json it works the same way incorrect.

The exception:
Failed to instantiate [com.google.cloud.storage.Storage]: Factory method 'googleCloudStorage' threw exception; nested exception is java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

The configuration class that is not able to instantiate credentials:

@Profile("prod")
@Configuration
public class GoogleCloudStorageConfiguration {

 @Bean
 @ConditionalOnProperty("spring.cloud.gcp.storage.enabled")
 public GoogleCloudStorageProperties googleCloudStorageProperties() {
   return new GoogleCloudStorageProperties();
 }

 @Bean
 @ConditionalOnBean(GoogleCloudStorageProperties.class)
 public Storage googleCloudStorage(CredentialsProvider credentials, GcpProjectIdProvider project) throws IOException {
   return StorageOptions.newBuilder()
       .setCredentials(credentials.getCredentials())
       .setProjectId(project.getProjectId())
       .setRetrySettings(RetrySettings.newBuilder().setMaxAttempts(3).build())
       .build()
       .getService();
 }
}

The part of service layer I use to read blobs:

public Resource get(String name) {
    var location = forFile(properties.bucket, name);
    var blob = new GoogleStorageResource(storage, location, false);
    LOGGER.info("[BLOB] Blob with [${}] successfully found.", name);
    try(var is = blob.getInputStream()) {
      return new InputStreamResource(is);
    } catch (IOException exception) {
      throw new UnexpectedException(exception.getMessage());
    }
  }