Gapic::UniverseDomainMismatch errors after updating to googleauth 1.12.0

[tartakynov]

After updating the googleauth gem from 1.11.2 to version 1.12.0, we're getting Gapic::UniverseDomainMismatch errors when trying to access BigTable.

The only documented change in this version was the addition of an optional debug logger. It seemed strange that this minor change would cause the error, so I looked into the code for other differences. I discovered that the update stopped fetching the access token (see here).

I tested if eagerly fetching the access token would fix the issue

project = Google::Cloud::Bigtable.new(**options)
credentials = project.service.credentials
if credentials.respond_to?(:client)
  client = credentials.client
  client.fetch_access_token! if client.needs_access_token?
end

And indeed this seems to fix the issue - no more Gapic::UniverseDomainMismatch errors.

Libraries used:

• google-cloud-bigtable-v2 (1.3.0)
• google-cloud-bigtable (2.11.1)
• googleauth (1.12.0)

[tartakynov]

Could be related to #504

[dazuma]

Thanks. I'm looking at it now.

[dazuma]

@tartakynov Which call is raising the exception? (Which method are you calling in the bigtable client?) Can you post a full stack trace?

[tartakynov]

@dazuma here's the partial stacktrace

gems/gapic-common-0.21.1/lib/gapic/universe_domain_concerns.rb:68:in `setup_universe_domain': Universe domain is googleapis.com but credentials are in  (Gapic::UniverseDomainMismatch)
    from gems/gapic-common-0.21.1/lib/gapic/grpc/service_stub.rb:78:in `initialize'
    from gems/google-cloud-bigtable-v2-1.3.0/lib/google/cloud/bigtable/v2/bigtable/client.rb:171:in `new'
    from gems/google-cloud-bigtable-v2-1.3.0/lib/google/cloud/bigtable/v2/bigtable/client.rb:171:in `initialize'
    from gems/google-cloud-bigtable-2.11.1/lib/google/cloud/bigtable/service.rb:896:in `new'
    from gems/google-cloud-bigtable-2.11.1/lib/google/cloud/bigtable/service.rb:896:in `create_bigtable_client'
    from gems/google-cloud-bigtable-2.11.1/lib/google/cloud/bigtable/service.rb:99:in `block in client'
    from gems/google-cloud-bigtable-2.11.1/lib/google/cloud/bigtable/service.rb:97:in `synchronize'
    from gems/google-cloud-bigtable-2.11.1/lib/google/cloud/bigtable/service.rb:97:in `client'
    from gems/google-cloud-bigtable-2.11.1/lib/google/cloud/bigtable/table.rb:81:in `initialize'
    from gems/google-cloud-bigtable-2.11.1/lib/google/cloud/bigtable/table.rb:679:in `new'
    from gems/google-cloud-bigtable-2.11.1/lib/google/cloud/bigtable/table.rb:679:in `from_path'
    from gems/google-cloud-bigtable-2.11.1/lib/google/cloud/bigtable/project.rb:350:in `table'

The code that raises looks like this

project = Google::Cloud::Bigtable.new(**options)
project.table(instance_id, table_name)

[dazuma]

@tartakynov Thanks! First, I suggest updating your gapic-common gem to version 0.23.0 or later. That might resolve this.

If it doesn't resolve the problem, can you provide the following:

• What kind of credential are you using (in your example code at the top, credentials.class and client.class)?
• If you are running on a Google compute platform (such as Google Cloud Run or Google Kubernetes Engine), which platform are you running on?
• Are you setting any credentials-related environment variables (such as GOOGLE_APPLICATION_CREDENTIALS, BIGTABLE_KEYFILE, etc.)?

[tartakynov]

@dazuma Unfortunately we can't upgrade to gapic-common 0.23.0 yet because we're locked into grpc < 1.65 for now and gapic-common versions 0.23.0 and onwards depend on grpc >= 1.65

[dazuma]

@tartakynov I see. Unfortunately I don't think we can loosen the grpc dependency, due to protobuf compatibility issues.

Your workaround may continue to work for now, though it's still puzzling me a bit why this is happening, and I'd love to get to the bottom of it.

Do you have answers to the three questions I posted above a few minutes ago?

Also, just to make sure, can you check that the GOOGLE_CLOUD_UNIVERSE_DOMAIN environment variable isn't set? (In particular, that it's not set as opposed to being set to the empty string.)

(EDIT: nvm that last item; your stacktrace implies that it is indeed not set.)

[tartakynov]

Just confirmed that updating gapic-common gem to 0.24.0 resolves the issue. But we can't do that in production due to the issue I mentioned earlier

@dazuma here are the answers to your questions

What kind of credential are you using (in your example code at the top, credentials.class and client.class)?

Google::Cloud::Bigtable::Credentials and Google::Auth::GCECredentials

If you are running on a Google compute platform (such as Google Cloud Run or Google Kubernetes Engine), which platform are you running on?

Google Kubernetes Engine

Are you setting any credentials-related environment variables (such as GOOGLE_APPLICATION_CREDENTIALS, BIGTABLE_KEYFILE, etc.)?

We're not setting any credentials-related environment variables

Thank you for looking into this 🙇

[dazuma]

Thanks for your help on this! We have a fix (#509) that we will try to get reviewed and pushed out in the next few hours.

[dazuma]

@tartakynov Okay, googleauth 1.12.2 is released. Let me know if it fixes your issue. (Feel free to reopen if it doesn't.)

[tartakynov]

@dazuma confirm that version 1.12.2 fixes the issue for us. Thank you for promptly resolving this! 🙇