Reproducible Builds

[IzzySoft]

I've checked your app if its build is reproducible (see: Reproducible bulds, special client support and more in our repo), but while I was able to successfully generate the APK by manually specifying the Flutter version (having flutter as a submodule in your git repo would eliminate the manual step here), the differences to the one provided at your latest release were huge. Which is at least partly due to Flutter embedding the build paths into its native libs (could you please let me know the path you are building in?) – but seemingly the APK here was also build from a commit other than the tag points to.

Here's the APK diff:

-------------------------------
--- /dev/fd/63  2024-07-21 15:17:01.987148320 +0200
+++ /dev/fd/62  2024-07-21 15:17:01.987148320 +0200
@@ -1,19 +1,19 @@
   META-INF/com/android/build/gradle/app-metadata.properties
   32-bit CRC value (hex):                         7e113144
   assets/dexopt/baseline.prof
-  32-bit CRC value (hex):                         ef5a1df3
+  32-bit CRC value (hex):                         0842fa9b
   assets/dexopt/baseline.profm
   32-bit CRC value (hex):                         d828a793
   classes.dex
-  32-bit CRC value (hex):                         795b3175
+  32-bit CRC value (hex):                         0f4511f0
   classes2.dex
   32-bit CRC value (hex):                         57b2ecfe
   lib/arm64-v8a/libapp.so
-  32-bit CRC value (hex):                         776093eb
+  32-bit CRC value (hex):                         1ab192aa
   lib/arm64-v8a/libc++_shared.so
   32-bit CRC value (hex):                         da048360
   lib/arm64-v8a/libflutter.so
-  32-bit CRC value (hex):                         64fb2801
+  32-bit CRC value (hex):                         ff384be5
   lib/arm64-v8a/libjniPdfium.so
   32-bit CRC value (hex):                         5abf3f4c
   lib/arm64-v8a/libmodft2.so
@@ -33,7 +33,7 @@
   assets/flutter_assets/FontManifest.json
   32-bit CRC value (hex):                         f4e36024
   assets/flutter_assets/NOTICES.Z
-  32-bit CRC value (hex):                         3cd547d7
+  32-bit CRC value (hex):                         b4cfcf15
   assets/flutter_assets/assets/captcha.svg
   32-bit CRC value (hex):                         dbe919e2
   assets/flutter_assets/assets/delete_confirmation.svg
@@ -2469,7 +2469,7 @@
   res/z3.xml
   32-bit CRC value (hex):                         cc6ffadf
   res/z6
-  32-bit CRC value (hex):                         f987d267
+  32-bit CRC value (hex):                         37103375
   res/zH.xml
   32-bit CRC value (hex):                         71337847
   res/zO.xml

As pointed out above, the diff in the *.so can most likely be eliminated if we use the same build path here that you have. I gladly try another run if you can provide me the mentioned details (build path and commit hash).

We'd appreciate if you could help making your build reproducible. We've prepared some hints on reproducible builds for that.

Looking forward to your reply!

[IzzySoft]

@dstark5 you're still around?

[dstark5]

Yes @IzzySoft sorry for the delayed response

[droidjedininja]

Hey, are you guys going to fix ( no available mirrors )? I can't download a single book?

[IzzySoft]

@dstark5 thanks for your response! Do you need additional details from our end (for the differing dex)?

@droidjedininja please don't hijack issues not related to your question 😉 It would be much preferred if you could open your own issue.

[droidjedininja]

Sorry bout that. Wasn't trying to hi-jack your space. I opened my own Issue, thanks.

[IzzySoft]

Thanks @droidjedininja! Helps to keep things clear 😉

@dstark5 could it be you build on Windows? libapp.so e.g. contains a string file:///C:/Users/rog/StudioProjects/Libgen/.dart_tool/flutter_build/dart_plugin_registrant.dart…

[dstark5]

Hey, are you guys going to fix ( no available mirrors )? I can't download a single book?

Hey Hi👋,
The issue will be fixed within a couple of days, Thank you

[dstark5]

Thanks @droidjedininja! Helps to keep things clear 😉

@dstark5 could it be you build on Windows? libapp.so e.g. contains a string file:///C:/Users/rog/StudioProjects/Libgen/.dart_tool/flutter_build/dart_plugin_registrant.dart…

Yes @IzzySoft

[IzzySoft]

Aw… OK, guess then we have no chance to match the paths – and thus cannot achieve reproducible builds. Just to make sure I didn't miss something: cc @obfusk

[obfusk]

You could try setting build_repo_dir: /C:/Users/rog/StudioProjects/Libgen and build_home_dir: /C:/Users/rog. It could work if the embedded paths are all file: URLs.

[IzzySoft]

Nope, unfortunately not:

useradd: invalid home directory '/C:/Users/rog'

[obfusk]

useradd: invalid home directory '/C:/Users/rog'

Ah. That would require some adjustments to provisioning then. But you could try android_home: /C:/Users/rog/sdk, build_repo_dir: /C:/Users/rog/StudioProjects/Libgen and keep build_home_dir as usual.

[IzzySoft]

Sorry, that doesn't work either:

+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/C:/Users/rog/sdk/cmdline-tools/12.0/bin
+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/C:/Users/rog/sdk/cmdline-tools/12.0/bin
+ yes
+ sdkmanager --sdk_root=/C:/Users/rog/sdk --licenses
/scripts/provision.sh: line 30: sdkmanager: command not found
+ true

Colon is the separator in PATH, so android_home gets split into two separate entries there.

[IzzySoft]

@dstark5 I've just chatted with Fay to see what options are left. We tried some more – but from our end, we can't get it built reproducibly; the only remaining option would be if you could build on Linux. You're welcome to use Fay's rbtlog for that (we could give you our build recipe for that to get you started), which would almost guarantee RB then.

As RB is not mandatory at IzzyOnDroid (though highly recommended), we leave the decision to you and of course accept it if you say that's asked too much. Just let us know please what you decide.

Thanks a lot for having us supported up to here!

[dstark5]

Hi @IzzySoft 👋
I've mistakenly changed the app version build number to 1.0.7+1 which haven't triggered any build on the izzyOnDroid now I have changed the build number to +10 but still the build isn't triggered

[IzzySoft]

As it isn't set up for RB, there are no builds triggered here (the APKs for the repo are taken directly from your releases, signed by you). So what build do you mean? If for RB, until set up that's done manually. And we'd need an APK from you plus the matching tag/commit it was built from.

[IzzySoft]

I've just tried to build v3.0.9-beta, but failed:

The current Dart SDK version is 3.2.3.

Because openlib requires SDK version >=3.3.0 <4.0.0, version solving failed.

The Flutter version I used here was 3.16.4. OK, switched the recipe to 3.24.4, that seems to work so far (would help if you'd include the Flutter version to use as git submodule, so it doesn't need manual adjustments every next version). Build succeeds, but RB fails the very same way as above (except for the NOTICES.Z file).

So I guess we have no chance here – as we cannot get the embedded path names fixed?

  -rw-r--r--  0.0 unx       56 b-       52 defN 1981-01-01 01:01:02 7e113144 META-INF/com/android/build/gradle/app-metadata.properties
- -rw-r--r--  0.0 unx     2403 b-     2403 stor 1981-01-01 01:01:02 95cf7338 assets/dexopt/baseline.prof
+ -rw-r--r--  0.0 unx     2403 b-     2403 stor 1981-01-01 01:01:02 d2004a77 assets/dexopt/baseline.prof
  -rw-r--r--  0.0 unx      240 b-      240 stor 1981-01-01 01:01:02 b1922ab2 assets/dexopt/baseline.profm
- -rw-r--r--  0.0 unx  7630296 b-  2745438 defN 1981-01-01 01:01:02 e6b9f4c4 classes.dex
- -rw-r--r--  0.0 unx  8520608 b-  3465630 defN 1981-01-01 01:01:02 8f31006c lib/arm64-v8a/libapp.so
+ -rw-r--r--  0.0 unx  7630296 b-  2745438 defN 1981-01-01 01:01:02 9090816e classes.dex
+ -rw-r--r--  0.0 unx  8520608 b-  3459607 defN 1981-01-01 01:01:02 6ef9ecc0 lib/arm64-v8a/libapp.so
  -rw-r--r--  0.0 unx  1022136 b-   304823 defN 1981-01-01 01:01:02 da048360 lib/arm64-v8a/libc++_shared.so
- -rw-r--r--  0.0 unx 10714448 b-  4990747 defN 1981-01-01 01:01:02 d7d3055d lib/arm64-v8a/libflutter.so
+ -rw-r--r--  0.0 unx 10714752 b-  4991242 defN 1981-01-01 01:01:02 9f62004e lib/arm64-v8a/libflutter.so
  -rw-r--r--  0.0 unx    53336 b-    21511 defN 1981-01-01 01:01:02 5abf3f4c lib/arm64-v8a/libjniPdfium.so
  -rw-r--r--  0.0 unx   554880 b-   276746 defN 1981-01-01 01:01:02 23f3bea1 lib/arm64-v8a/libmodft2.so
  -rw-r--r--  0.0 unx  5216024 b-  2426621 defN 1981-01-01 01:01:02 c760a7d5 lib/arm64-v8a/libmodpdfium.so
@@ -14,7 +14,7 @@
  -rw-r--r--  0.0 unx     1657 b-      381 defN 1981-01-01 01:01:02 00b04e14 assets/flutter_assets/AssetManifest.bin
  -rw-r--r--  0.0 unx     1509 b-      339 defN 1981-01-01 01:01:02 01a59096 assets/flutter_assets/AssetManifest.json
  -rw-r--r--  0.0 unx      208 b-      111 defN 1981-01-01 01:01:02 f4e36024 assets/flutter_assets/FontManifest.json
- -rw-r--r--  0.0 unx   108554 b-   103657 defN 1981-01-01 01:01:02 5fcc40b5 assets/flutter_assets/NOTICES.Z
+ -rw-r--r--  0.0 unx   108195 b-   103351 defN 1981-01-01 01:01:02 249cc23e assets/flutter_assets/NOTICES.Z
  -rw-r--r--  0.0 unx    10813 b-     4463 defN 1981-01-01 01:01:02 dbe919e2 assets/flutter_assets/assets/captcha.svg
  -rw-r--r--  0.0 unx     8201 b-     3260 defN 1981-01-01 01:01:02 0a847596 assets/flutter_assets/assets/delete_confirmation.svg
  -rw-r--r--  0.0 unx     8725 b-     4258 defN 1981-01-01 01:01:02 f271cebe assets/flutter_assets/assets/empty_mylib.svg
@@ -494,14 +494,11 @@
  -rw----     0.0 fat     1171 b-     1171 stor 1981-01-01 01:01:02 0d1f216a res/yq.png
  -rw----     0.0 fat      424 b-      199 defN 1981-01-01 01:01:02 3b16abbc res/z1.xml
  -rw----     0.0 fat      396 b-      228 defN 1981-01-01 01:01:02 cc6ffadf res/z3.xml
- -rw----     0.0 fat     1996 b-     1452 defN 1981-01-01 01:01:02 f987d267 res/z6
+ -rw----     0.0 fat     1964 b-     1426 defN 1981-01-01 01:01:02 37103375 res/z6
  -rw----     0.0 fat     1116 b-      523 defN 1981-01-01 01:01:02 ee714821 res/zH.xml

[dstark5]

@IzzySoft what should I need to fix this issues

[IzzySoft]

As long as we cannot fix the "embedded path" issue, I don't think it makes sense to address the other issues (no need to put unnecessary burden upon you). As we already tried what we can, the only way out of that I currently see is you building the release APKs on Linux instead of Windows. You could e.g. use rbtlog for that, it can be used to "just build the APK and give it to me". We could give you the "recipe file" we used here so you don't have to try-and-err setting that up. And I wouldn't be surprised if the other differences (NOTICES.Z, res/z6 – and with them classes.dex and baseline.prof) would simply disappear as well.

Altternatively, a Github workflow building on e.g. Debian or Ubuntu should take care for that as well. In both cases, all that's left is your pulling the APK to sign it, then attach it to releases.

If that's asking too much: no bad feelings. We'd simply have to give up then. But RB is not mandatory at IzzyOnDroid – so while being sad, it won't be a show-stopper. RB is just another security feature so people can be confident the APK is really reflecting what it's claimed to be, built from the very source indicated.

[dstark5]

@IzzySoft will do the things you've mentioned. Why 1.0.9 build isn't in izzydroid

[IzzySoft]

Let me check. Looks like IoD grew so much and most apps are on Github, the updater might be running into API limits – though the logs down't show such here… yupp, manually triggering an update check and it is found. This is the 3rd app now I see this happen to 😢 Wonder why there's nothing in the logs. Well, looks like the pile on my desk is growing…

Thanks – and the update will show up with the next sync around 6 pm UTC.

PS: the badge on the readme points to android.izzysoft.de/repo/apk/com.app.openlib, which is (still) only a secondary, much limited browser (another pile on my desk). Could you adjust that to apt.izzysoft.de/packages/com.app.openlib, which is on the primary repo browser?

[IzzySoft]

Ah, btw:

2024-10-20 14:10:05 local ERROR  ! repo/com.app.openlib_2012.apk declares sensitive permission(s):
           android.permission.READ_MEDIA_IMAGES android.permission.READ_MEDIA_AUDIO
2024-10-20 14:10:06 local ERROR  ! repo/com.app.openlib_2012.apk contains signature block blobs: 0x504b4453 (DEPENDENCY_INFO_BLOCK; GOOGLE)

Could you please clarify what those 2 permissions are needed/used for? As for DEPENDENCY_INFO_BLOCK, easily avoided with a minor addition to your build.gradle:

android {
    dependenciesInfo {
        // Disables dependency metadata when building APKs.
        includeInApk = false
        // Disables dependency metadata when building Android App Bundles.
        includeInBundle = false
    }
}

For some background: that BLOB is supposed to be just a binary representation of your app's dependency tree. But as it's encrypted with a public key belonging to Google, only Google can read it – and nobody else can even verify what it really contains. More details can be found e.g. here: Ramping up security: additional APK checks are in place with the IzzyOnDroid repo.

[dstark5]

Hi @IzzySoft
The Read image and audio permissions are for the storage permission on android version above 12 this permission is only asked if the user need to store books on internal storage.

Will add the additions to build.gradle as you've mentioned

[IzzySoft]

permission is only asked if the user need to store books on internal storage.

Wait: you need to ask READ permissions for your own files? As I read the explanations of scoped storage, that would only be needed to read media your app doesn't "own" (i.e. created itself). Did I get that wrong? Forgive me then, but not being an Android dev I've never implemented anything along those lines, so I lack first-hand experience.

Will add the additions to build.gradle as you've mentioned

Thanks!

[dstark5]

@IzzySoft

https://stackoverflow.com/questions/73985513/android-permission-read-media-audio-and-read-external-storage-for-api-level-33

Pls take a look and correct me if I'm wrong

Also izzyOnDroid doesn't triggered build for new version can you pls take a look at the issue

[IzzySoft]

Pls take a look and correct me if I'm wrong

That's too unspecific. It doesn't ask about scoped storage, but about read access to storage in general (if I did not miss anything). In other words, author asks what permission they need to read media files from storage – but not if any such permission is required for the files created by the app itself. So let's take a look at the official documentation:

Whether your app needs permissions to access storage depends on whether it accesses only its own media files or files created by other apps.

So I remembered that one correctly. What does that documentation say to "Access your own media files"?

On devices that run Android 10 or higher, you don't need storage-related permissions to access and modify media files that your app owns, including files in the MediaStore.Downloads collection. If you're developing a camera app, for example, you don't need to request storage-related permissions to access the photos it takes, because your app owns the images that you're writing to the media store.

According to the app description, you only need access to the files OpenLib downloaded itself, right? Then you won't need the Android 11+ permissions (READ_MEDIA_*), but might need READ_EXTERNAL_STORAGE for Android < 11 if the files are stored outside the app's own scope (i.e. outside /data/data/com.app.openlib or its pendant on external storage). You might however need them if OpenLib should be able to access "other apps' files", but then this should be pointed out.

---

izzyOnDroid doesn't triggered build for new version can you pls take a look at the issue

UpdateCheckMode: Static

That means "monthly checks". Now let's see why, in the MaintainerNotes:

- 2024-09-02 UCM Tags=>Static as latest release has VC decreased and thus is daily pulled and deleted right away – https://github.com/dstark5/Openlib/issues/63#issuecomment-2325240850

There was no response at my question yet, hence it's still on "Static" 🤷‍♂️ And as there must have been 2 checks already meanwhile, and no update showed up at IoD, the issue probably still exists. I see you've updated pubspec.yaml accordingly and then created a new release yesterday afternoon, which was after the last check. So I can manually trigger an update to check and, if it fits, re-enable the daily checks, give me a second…

$ iod repo get com.app.openlib
com.app.openlib: looking for 'https://api.github.com/repos/dstark5/Openlib/releases'
com.app.openlib: checking tag 'v1.0.10-beta'
com.app.openlib: lastRelNo set to 'v1.0.10-beta', checking for files
com.app.openlib: Upstream file date (2024-11-02 12:08) is newer than ours (2024-10-20 14:14).
com.app.openlib: returning ['v1.0.10-beta','https://github.com/dstark5/Openlib/releases/download/v1.0.10-beta/openlib-arm64-v8a-release.apk',1730545712]
com.app.openlib: 1.0.9/v1.0.10-beta, https://github.com/dstark5/Openlib/releases: https://github.com/dstark5/Openlib/releases/download/v1.0.10-beta/openlib-arm64-v8a-release.apk
- Grabbing update for com.app.openlib: OK
- Checking 'repo/com.app.openlib_2013.apk' for libraries and malware …
- Checking the app's AndroidManifest.xml …
! repo/com.app.openlib_2013.apk declares sensitive permission(s): android.permission.READ_MEDIA_IMAGES android.permission.READ_MEDIA_AUDIO
...

OK, that seems to look fine now: repo dir currently holds 2011, 2012 and now 2013, enabling daily checks again. v1.0.10 should become available with the next sync around 7 pm UTC.

[IzzySoft]

I'd give the RB another try if you could tell me which path you built from (Flutter embeds paths into its libs, so they must match).