Proguard/R8 is stripping out fields from generated file class AesGcmKeyFormat #361

simtse · 2020-05-20T00:48:29Z

Coming from Tink 1.3.0 stable to Tink 1.4 RC2 (on Android), I noticed that the AesGcmKeyFormat has a possible problem with proguard/R8. We haven't added anything in our proguard-rules.pro file for Tink and am getting a stack trace like this

java.lang.NoClassDefFoundError: com.google.crypto.tink.aead.AeadKeyTemplates
	... 13 more
Caused by: java.lang.NoClassDefFoundError: com.google.crypto.tink.aead.AeadKeyTemplates
	at slack.commons.security.AeadPrimitiveFactoryImpl.getAeadPrimitive(AeadPrimitiveFactory.kt:4)
	... 12 more
Caused by: java.lang.ExceptionInInitializerError
	at androidx.security.crypto.EncryptedSharedPreferences$PrefValueEncryptionScheme.
<clinit>(EncryptedSharedPreferences.java:1)
        ... 11 more
Caused by: java.lang.RuntimeException: Field version_ for com.google.crypto.tink.proto.AesGcmKeyFormat not found. Known fields are [public int com.google.crypto.tink.proto.AesGcmKeyFormat.keySize_, public static final com.google.crypto.tink.proto.AesGcmKeyFormat com.google.crypto.tink.proto.AesGcmKeyFormat.DEFAULT_INSTANCE, public static volatile com.google.crypto.tink.shaded.protobuf.Parser com.google.crypto.tink.proto.AesGcmKeyFormat.PARSER]
	at com.google.crypto.tink.shaded.protobuf.MessageSchema.reflectField(MessageSchema.java:7)
	at com.google.crypto.tink.shaded.protobuf.MessageSchema.newSchemaForRawMessageInfo(MessageSchema.java:53)
	at com.google.crypto.tink.shaded.protobuf.MessageSchema.newSchema(MessageSchema.java:2)
	at com.google.crypto.tink.shaded.protobuf.Protobuf.schemaFor(Protobuf.java:30)
	at com.google.crypto.tink.shaded.protobuf.Protobuf.schemaFor(Protobuf.java:48)
	at com.google.crypto.tink.shaded.protobuf.GeneratedMessageLite$Builder.buildPartial(GeneratedMessageLite.java:5)
	at com.google.crypto.tink.shaded.protobuf.GeneratedMessageLite$Builder.build(GeneratedMessageLite.java:1)
	at com.google.crypto.tink.aead.AeadKeyTemplates.createAesGcmKeyTemplate(AeadKeyTemplates.java:5)
	at com.google.crypto.tink.aead.AeadKeyTemplates.<clinit>(AeadKeyTemplates.java:1)

In the Android studio looking at the generated AesGcmKeyFormat, it has the _version

package com.google.crypto.tink.proto;

import ...;

public final class AesGcmKeyFormat extends GeneratedMessageLite<AesGcmKeyFormat, AesGcmKeyFormat.Builder> implements AesGcmKeyFormatOrBuilder {
    public static final int KEY_SIZE_FIELD_NUMBER = 2;
    private int keySize_;
    public static final int VERSION_FIELD_NUMBER = 3;
    private int version_;
    private static final AesGcmKeyFormat DEFAULT_INSTANCE;
    private static volatile Parser<AesGcmKeyFormat> PARSER;

...
}

But when looking at the build APK with proguard/R8 running through it, seems like version_ was removed

Seems like the AesSivKeyFormat has a similar problem as well.

I'm going to also check if this problem might have existed in the 1.3 release of the library. But I haven't seen this stack trace before.

We stricly use protobuf 3.12.0

+--- com.google.protobuf:protobuf-java:{strictly 3.12.0} -> 3.12.0 (c)

...

|    |    \--- com.google.protobuf:protobuf-javalite:3.12.0

...

|    |    +--- com.google.android.apps.common.testing.accessibility.framework:accessibility-test-framework:2.1
|    |    |    +--- org.hamcrest:hamcrest-core:1.3 -> org.hamcrest:hamcrest:2.2
|    |    |    \--- com.google.protobuf:protobuf-java:2.6.1 -> 3.12.0

The text was updated successfully, but these errors were encountered:

simtse · 2020-05-20T01:16:00Z

Checked with the 1.3.0 stable release that the AesGcmKeyFormat has the version_ and more members. It seems like proguard plays nicer with 1.3.

Same with the AesSivKeyFormat

simtse · 2020-05-20T04:26:52Z

Ahh in 1.3, we already had

-keep class * extends com.google.protobuf.GeneratedMessageLite { *; }

But since this is now in the shaded package, we added this to make it work

-keep class * extends com.google.crypto.tink.shaded.protobuf.GeneratedMessageLite { *; }

We definitely have other libraries that use protobuf in our project, I'm not sure how they conflict or cause problems.

tholenst · 2020-05-20T11:30:28Z

Apparently this is protocolbuffers/protobuf#6463. I will investigate more.

tholenst · 2020-05-20T11:32:36Z

(By the way, much thanks for the detailed report: this helps a lot).

AndroidX Security delegates crypto operation to Tink which depends on Protobuf Javalite. Recently Protobuf Javalite introduced a change that relies on reflection, which doesn't work with Proguard. This change adds a rule that keeps the (shaded) Protobuf classes in Tink as-is. See also: - https://buganizer.corp.google.com/issues/154315507 - tink-crypto/tink#361 - protocolbuffers/protobuf#6463 - https://b.corp.google.com/issues/144631039

thaidn · 2020-05-22T20:06:43Z

@simtse

Does this smaller rule work for you?

-keepclassmembers class * extends com.google.crypto.tink.shaded.protobuf.GeneratedMessageLite {
  <fields>;
}

thaidn · 2020-05-22T20:31:58Z

It looks like the right way to do this is to include this rule in Tink. According to https://developer.android.com/studio/build/shrink-code#configuration-files, we just need to put this rule in META-INF/proguard/.

GitHub issue: #361. PiperOrigin-RevId: 313043816

thaidn · 2020-05-25T09:03:28Z

@simtse HEAD-SNAPSHOT (and 1.4.0) includes a ProGuard file rule with the following rule:

-keepclassmembers class * extends com.google.crypto.tink.shaded.protobuf.GeneratedMessageLite {
  <fields>;
}

I believe your custom rule is no longer needed. Could you please test with HEAD-SNAPSHOT and confirm?

thaidn · 2020-05-25T18:04:38Z

I confirm that the fix works perfectly.

simtse · 2020-05-25T19:10:12Z

I'm running into some issues on my part so I'm doing some testing right now. I don't have anything conclusive yet, but this is an error that I get when taking HEAD-SNAPSHOT and removing my proguard rules.

java.lang.IncompatibleClassChangeError: com.google.crypto.tink.proto.AesCtrHmacAeadKey
at dalvik.system.DexFile.defineClassNative(Native Method)
at dalvik.system.DexFile.defineClass(DexFile.java:226)
at dalvik.system.DexFile.loadClassBinaryName(DexFile.java:219)
at dalvik.system.DexPathList.findClass(DexPathList.java:338)
at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:54)
at java.lang.ClassLoader.loadClass(ClassLoader.java:511)
at java.lang.ClassLoader.loadClass(ClassLoader.java:469)
at com.google.crypto.tink.aead.AesCtrHmacAeadKeyManager.<init>(AesCtrHmacAeadKeyManager.java:43)
at com.google.crypto.tink.aead.AeadConfig.<clinit>(AeadConfig.java:37)

And AndroidX EncryptedSharedPreferences has similar problem

java.lang.IncompatibleClassChangeError: com.google.crypto.tink.proto.AesSivKeyFormat
at dalvik.system.DexFile.defineClassNative(Native Method)
at dalvik.system.DexFile.defineClass(DexFile.java:226)
at dalvik.system.DexFile.loadClassBinaryName(DexFile.java:219)
at dalvik.system.DexPathList.findClass(DexPathList.java:338)
at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:54)
at java.lang.ClassLoader.loadClass(ClassLoader.java:511)
at java.lang.ClassLoader.loadClass(ClassLoader.java:469)
at com.google.crypto.tink.daead.DeterministicAeadKeyTemplates.createAesSivKeyTemplate(DeterministicAeadKeyTemplates.java:45)
at com.google.crypto.tink.daead.DeterministicAeadKeyTemplates.<clinit>(DeterministicAeadKeyTemplates.java:38)
at androidx.security.crypto.EncryptedSharedPreferences$PrefKeyEncryptionScheme.<clinit>(EncryptedSharedPreferences.java:146)

I'm still trying to figure out what might be the problem. Might be the emulator environment as well.

simtse · 2020-05-25T21:58:58Z

@thaidn, So this is going to sound odd as these are some combinations I've tried and their results.

HEAD-SNAPSHOT + removed local keep members proguard rule - ❌
HEAD-SNAPSHOT + with local keep class proguard rule - ❌
HEAD-SNAPSHOT + with local keep members proguard rule - ❌
1.4.0-rc2 + removed local keep members/class proguard rules - ❌ (this is expected as classes are stripped)
1.4.0-rc2 + using local keep class proguard rules - ✅
1.4.0-rc2 + using local keep memebers proguard rules - ✅

thaidn · 2020-05-25T23:24:07Z

Can you show me your build.gradle?

I suspect that this is because of caching. Can you remove $HOME/.gradle/.caches and try again?

simtse · 2020-05-25T23:34:29Z

Our build.gradle is kotlin script, so not the same for pasting Not sure what you're looking for specifically, so I'll just translate to what we have. The build is happening in a CI environment with a Google Cloud NexusLowRes device from Firebase test lab. I had seen this problem with IncompatibleClassChangeError before with Tink 1.3.0, but this stopped happening when I brought in Tink 1.4.0 RC2.

dependences {
  ...
  implementation 'com.google.crypto.tink:tink-android:HEAD-SNAPSHOT'
  ...
}

And we've declared the https://oss.sonatype.org/content/repositories/snapshots as a repository for the build.

Note: I updated my comment above #361 (comment) to say that ✅ worked when we declare the keep rules ourselves.

thaidn · 2020-05-25T23:53:32Z

Thanks @simtse. Have you tried removing $HOME/.gradle/.caches?

simtse · 2020-05-25T23:59:51Z

Will try locally with removing cache. Because this is running on Firebase test lab devices/emulators, I would have thought the caching doesn't really matter. But I'll try

simtse · 2020-05-26T00:00:07Z

Whoops sorry I hit the wrong button.

tytydraco · 2020-05-26T00:36:14Z

Thanks @simtse. Have you tried removing $HOME/.gradle/.caches?

I am having the issue stated above, with EncryptedSharedPrefs. I just invalidated my Android Studio and Gradle caches, cleaned my project, and restarted Android Studio. The issue persists when building a release APK with minify enabled. Here is my log if this helps:

Caused by: java.lang.RuntimeException: Field version_ for com.google.crypto.tink.proto.AesGcmKeyFormat not found. Known fields are [public int com.google.crypto.tink.proto.AesGcmKeyFormat.keySize_, public static final com.google.crypto.tink.proto.AesGcmKeyFormat com.google.crypto.tink.proto.AesGcmKeyFormat.DEFAULT_INSTANCE, public static volatile com.google.crypto.tink.shaded.protobuf.Parser com.google.crypto.tink.proto.AesGcmKeyFormat.PARSER]

thaidn · 2020-05-26T02:00:06Z

@tytydraco hi there, EncryptedSharedPrefs would require a different solution. It comes from AndroidX Security which depends on 1.4.0-rc2. You have two workarounds:

1/ Use this ProGuard rule

-keepclassmembers class * extends com.google.crypto.tink.shaded.protobuf.GeneratedMessageLite {
  <fields>;
}

2/ Configure Gradle to remove the 1.4.0-rc2 version and add a dependency on HEAD-SNAPSHOT.

Could you please try either way and let us know which one works for you?

tytydraco · 2020-05-26T02:09:50Z

Hey @thaidn, thanks for the help. Solution #1 solved the problem for me. I assume that in the future, I'll be able to remove the proguard rule. I appreciate your response.

thaidn · 2020-05-26T02:13:23Z

Hey @thaidn, thanks for the help. Solution #1 solved the problem for me. I assume that in the future, I'll be able to remove the proguard rule. I appreciate your response.

Thanks for confirming. Yeah the goal is to have the rule within Tink.

I added it to Tink in 8bdaed4. It works on my machine, but it seems that @simtse is having some trouble.

simtse · 2020-05-26T16:04:43Z

I'm working on this today again so I'll report back. I will try to build locally instead of in CI to see if it might be a CI caching problem.

simtse · 2020-05-26T21:20:50Z

Another weird thing happened when running locally with HEAD-SNAPSHOT locally with proguard. This doesn't happen if I depend on Tink 1.4.0-rc2. When using the newer API in building the AndroidKeysetManager

val keysetBuilder = AndroidKeysetManager.Builder()
          .withKeyTemplate(AesGcmKeyManager.aes256GcmTemplate())
          .withSharedPref(context, "keyset_name", "shared_prefs_file_name")
          .withMasterKeyUri("tink-key-alias")

Then got

Caused by: java.lang.NoSuchMethodError: No static method aes256GcmTemplate()Lcom/google/crypto/tink/KeyTemplate; in class Lcom/google/crypto/tink/aead/AesGcmKeyManager; or its super classes (declaration of 'com.google.crypto.tink.aead.AesGcmKeyManager' appears in base.apk)

However, if I were to use the now deprecated approach

.withKeyTemplate(AeadKeyTemplates.AES256_GCM)

I would be able to reference the field. The odd thing was that I also added the rule as well

-keep class * extends com.google.crypto.tink.shaded.protobuf.GeneratedMessageLite { *; }

but it still failed with the same error.
Locally, I haven't run into the incompatible class error. So it might be some problem in our caching and CI then.

It is odd that it consistently fails even after I cleared my caches from {HOME}/.gradle/.caches and also gradle commands to clean and build.

tholenst · 2020-05-27T07:17:00Z

Caused by: java.lang.NoSuchMethodError: No static method aes256GcmTemplate()Lcom/google/crypto/tink/KeyTemplate; in class Lcom/google/crypto/tink/aead/AesGcmKeyManager; or its super classes (declaration of 'com.google.crypto.tink.aead.AesGcmKeyManager' appears in base.apk)

I'm not sure that this is related to proguard though, it sounds like somehow an old version of AesGcmKeyManager made it in your compilation. How did you build it in this case?

simtse · 2020-05-27T17:37:54Z

@tholenst and @thaidn , yes so I figured out why I had those problems.
I had the AndroidX security library also, and it uses Tink 1.3.0 (or 1.4.0-rc4 for newer versions) and when pointing my exlicity dependency to HEAD-SNAPSHOT, it took the 1.3.0 version instead (or the one with the number) instead of HEAD-SNAPSHOT. This caused the builds to take an older API and appear to strip out code that wasn't in 1.3.0.

I updated the local builds and everything looks good now ✅

No incompatible class problems
No proguard stripping issues
Taking HEAD-SNAPSHOT over transitive 1.3.0 or 1.4.0-rc2 also consumes the new snapshot proguard rule 8bdaed4.

HEAD-SNAPSHOT looks good to fix the proguard issue with generated files from protobuf. 👍

thaidn · 2020-05-27T18:39:21Z

@simtse awesome, thank you!

Would you mind sharing your Gradle config, especially how you replace 1.3.0 in AndroidX Security with 1.4.0-rc2? I believe many would found it useful.

simtse · 2020-05-27T19:10:56Z

So my problem wasn't exactly getting AndroidX security to 1.4.0-rc2, I believe it was more that the HEAD-SNAPSHOT takes a lower precedence in dependency resolution if ordered by alphabetically?

I had something like this in Gradle config (kotlin gradle script). But the groovy implementation is very similar.

dependences {
  ...
  implementation("com.google.crypto.tink:tink-android:HEAD-SNAPSHOT")
  implementation("androidx.security:security-crypto:1.0.0-rc01") // uses Tink 1.3.0
  ...
}

And for whatever reason, it would pick up 1.3.0 Tink when building to the APK. So I had to override resolution strategy to get HEAD-SNAPSHOT

configurations.all {
  resolutionStrategy.eachDependency {
    if (requested.group == "com.google.crypto.tink" && requested.name == "tink-android" && (requested.version != "HEAD-SNAPSHOT")) {
      useVersion("HEAD-SNAPSHOT")
      because("Testing out snapshots")
    }
  }
}

thaidn · 2020-05-28T17:14:49Z

This is very helpful, thanks @simtse.

I'll leave this open until 1.4.0 is released.

thaidn · 2020-07-14T21:29:48Z

https://github.com/google/tink/releases/tag/v1.4.0 was released a few minutes ago.

michael-behnam-carriage · 2023-05-02T13:20:30Z

Hi @thaidn
i tried the above solution but it didn't work for me
also i tried different versions for tink library and i have the same problem

implementation 'androidx.security:security-crypto:1.1.0-alpha03'
implementation 'com.google.crypto.tink:tink-android:1.5.0'

i can find tink-android:1.5.0 and tink:1.3.0-rc2 in the gradle cache

@tholenst i wonder if you can help aslo

simtse changed the title ~~Android Proguard/R8 is stripping out fields from generated file class AesGcmKeyFormat~~ Proguard/R8 is stripping out fields from generated file class AesGcmKeyFormat May 20, 2020

tholenst self-assigned this May 20, 2020

thaidn mentioned this issue May 22, 2020

Add a Proguard rule to keep Protobuf classes as-is. tfcporciuncula/androidx-security-issue#1

Open

thaidn added a commit that referenced this issue May 25, 2020

Add a ProGuard/R8 rule to keep the (shaded) Protobuf classes as-is.

8bdaed4

GitHub issue: #361. PiperOrigin-RevId: 313043816

simtse closed this as completed May 25, 2020

simtse reopened this May 25, 2020

JeroenMols mentioned this issue Jun 10, 2020

Caused by java.lang.RuntimeException with proguard rules enabled. plaid/plaid-link-android#76

Closed

thaidn mentioned this issue Jun 17, 2020

Protobuf related build issue with Tink + Firebase (Android) #301

Closed

thaidn closed this as completed Jul 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proguard/R8 is stripping out fields from generated file class AesGcmKeyFormat #361

Proguard/R8 is stripping out fields from generated file class AesGcmKeyFormat #361

simtse commented May 20, 2020 •

edited

Loading

simtse commented May 20, 2020 •

edited

Loading

simtse commented May 20, 2020 •

edited

Loading

tholenst commented May 20, 2020

tholenst commented May 20, 2020

thaidn commented May 22, 2020

thaidn commented May 22, 2020

thaidn commented May 25, 2020

thaidn commented May 25, 2020

simtse commented May 25, 2020 •

edited

Loading

simtse commented May 25, 2020 •

edited

Loading

thaidn commented May 25, 2020 •

edited

Loading

simtse commented May 25, 2020

thaidn commented May 25, 2020

simtse commented May 25, 2020

simtse commented May 26, 2020

tytydraco commented May 26, 2020

thaidn commented May 26, 2020

tytydraco commented May 26, 2020

thaidn commented May 26, 2020

simtse commented May 26, 2020

simtse commented May 26, 2020 •

edited

Loading

tholenst commented May 27, 2020

simtse commented May 27, 2020 •

edited

Loading

thaidn commented May 27, 2020 •

edited

Loading

simtse commented May 27, 2020 •

edited

Loading

thaidn commented May 28, 2020

thaidn commented Jul 14, 2020

michael-behnam-carriage commented May 2, 2023 •

edited

Loading

Proguard/R8 is stripping out fields from generated file class AesGcmKeyFormat #361

Proguard/R8 is stripping out fields from generated file class AesGcmKeyFormat #361

Comments

simtse commented May 20, 2020 • edited Loading

simtse commented May 20, 2020 • edited Loading

simtse commented May 20, 2020 • edited Loading

tholenst commented May 20, 2020

tholenst commented May 20, 2020

thaidn commented May 22, 2020

thaidn commented May 22, 2020

thaidn commented May 25, 2020

thaidn commented May 25, 2020

simtse commented May 25, 2020 • edited Loading

simtse commented May 25, 2020 • edited Loading

thaidn commented May 25, 2020 • edited Loading

simtse commented May 25, 2020

thaidn commented May 25, 2020

simtse commented May 25, 2020

simtse commented May 26, 2020

tytydraco commented May 26, 2020

thaidn commented May 26, 2020

tytydraco commented May 26, 2020

thaidn commented May 26, 2020

simtse commented May 26, 2020

simtse commented May 26, 2020 • edited Loading

tholenst commented May 27, 2020

simtse commented May 27, 2020 • edited Loading

thaidn commented May 27, 2020 • edited Loading

simtse commented May 27, 2020 • edited Loading

thaidn commented May 28, 2020

thaidn commented Jul 14, 2020

michael-behnam-carriage commented May 2, 2023 • edited Loading

simtse commented May 20, 2020 •

edited

Loading

simtse commented May 20, 2020 •

edited

Loading

simtse commented May 20, 2020 •

edited

Loading

simtse commented May 25, 2020 •

edited

Loading

simtse commented May 25, 2020 •

edited

Loading

thaidn commented May 25, 2020 •

edited

Loading

simtse commented May 26, 2020 •

edited

Loading

simtse commented May 27, 2020 •

edited

Loading

thaidn commented May 27, 2020 •

edited

Loading

simtse commented May 27, 2020 •

edited

Loading

michael-behnam-carriage commented May 2, 2023 •

edited

Loading