F-Droid reproduceable build failed #1639

linsui · 2022-04-21T05:35:05Z

Describe the bug

https://monitor.f-droid.org/builds/log/org.jellyfin.androidtv/130399#site-footer It built fine in the CI. Not sure why it failed on the build server. Could you please take a look? Thanks!

Logs

https://monitor.f-droid.org/builds/log/org.jellyfin.androidtv/130399#site-footer

Application version

130399

Where did you install the app from?

No response

Device information

None

Android version

None

Jellyfin server version

None

sourpatched · 2022-04-21T12:40:42Z

Seems like it could be due to APK Signature that's causing a mismatch?

F-Drod is more for mobile then ATV but in saying that, it's best to have an app be more accessible.

nielsvanvelzen · 2022-05-07T09:12:52Z

Issue might be fixed in the v0.13.4 release.

linsui · 2022-05-08T15:21:47Z

Still not verified.

linsui · 2022-05-31T08:04:27Z

There are two different files.

Unexpected diff output:
Binary files /tmp/tmpn4i7qii7/unsigned_binaries_org.jellyfin.androidtv_130699.binary/content/assets/dexopt/baseline.prof and /tmp/tmpn4i7qii7/_tmp_tmpn4i7qii7_sigcp_org.jellyfin.androidtv_130699/content/assets/dexopt/baseline.prof differ
Binary files /tmp/tmpn4i7qii7/unsigned_binaries_org.jellyfin.androidtv_130699.binary/content/classes3.dex and /tmp/tmpn4i7qii7/_tmp_tmpn4i7qii7_sigcp_org.jellyfin.androidtv_130699/content/classes3.dex differ

licaon-kter · 2022-08-03T08:49:50Z

Still an issue :( https://gitlab.com/fdroid/fdroiddata/-/jobs/2806346970#L636

Disabled autoupdate for now: https://gitlab.com/fdroid/fdroiddata/-/commit/34bd6224241835aa3e761e02c1db0f00c3aca955

nielsvanvelzen · 2022-08-03T19:39:30Z

Thanks for linking the job, I think I know why it fails now. The AboutLibraries plugin we use generates a JSON file with all licenses and adds a timestamp from when it generated the file. That timestamp obviously differs between our build and F-Droid's build. Will see what I can do to fix.

nielsvanvelzen · 2022-08-05T06:41:08Z

Issue should be fixed in the next release (0.14.1)

licaon-kter · 2022-09-15T07:54:02Z

Still is failing locally: 2 files differ, can you dissect the APK? (remove .zip at the end of the filenames and use 7zip to unpack)

jtv.7z.001.zip
jtv.7z.002.zip
jtv.7z.003.zip
jtv.7z.004.zip

ref: https://gitlab.com/linsui/fdroiddata/-/commit/62ec23b72ffea96297c82641fbbfa85f162b7b1e#note_1100650641

linsui · 2023-01-09T16:37:54Z

With

tasks.whenTaskAdded {
    if(name.contains("ArtProfile")) {
        enabled = false
    }
}

we can disable the prefile. Not sure if it affects the performance.

nielsvanvelzen · 2023-01-09T20:16:45Z

I'm quite sure that disabling ART will give us a performance penalty, so we shouldn't disable it. If that's the problem in creating a reproducible build I think our only choice is to not use reproducible builds for now.

obfusk · 2023-01-10T05:38:50Z

I'm quite sure that disabling ART will give us a performance penalty, so we shouldn't disable it.

It doesn't disable the Android Runtime (ART), just baseline profiles.

According to the docs:

Baseline Profiles improve code execution speed by around 30% from the first launch by avoiding interpretation and just-in-time (JIT) compilation steps for included code paths.

So it may very well indeed result in a performance penalty, but I have found no real-world data on whether 30% is accurate or not. And all the docs say that developers need to define these profiles, but I don't see e.g. a baseline-prof.txt in your repo, so I have no idea what the baseline.prof{,m} is currently based on and whether it's actually doing anything useful. Do you have any info on that?

linsui · 2023-01-10T06:11:21Z

Could you please test the apk build without the profile?

linsui · 2023-01-10T11:07:24Z

Every time it produces a different apk so the profile part is not reproducible at all.

linsui · 2023-01-11T09:18:45Z

@licaon-kter got the 0.14.3 version build reproduciblly by limiting it to use only 1 core.

@nielsvanvelzen Do you use more cores for newer versions? Could you please build the latest version with 1 core?

nielsvanvelzen · 2023-01-11T10:41:46Z

All releases are build with GitHub actions. According to the documentation these are the specs:

2-core CPU (x86_64)
7 GB of RAM
14 GB of SSD space

I believe they used two cores from the start. Our 0.14 releases also used this release pipeline.

linsui · 2023-01-16T05:19:23Z

@obfusk's https://github.com/obfusk/reproducible-apk-tools#sort-baselinepy can fix the profm problem. You also need to run the script before sign the apk.

obfusk · 2023-01-20T13:41:13Z

You can also sort the baseline.profm file by adding something like this to build.gradle:

import com.android.tools.profgen.ArtProfileKt
import com.android.tools.profgen.ArtProfileSerializer
import com.android.tools.profgen.DexFile

project.afterEvaluate {
    tasks.compileReleaseArtProfile.doLast {
        outputs.files.each { file ->
            if (file.toString().endsWith(".profm")) {
                println("Sorting ${file} ...")
                def version = ArtProfileSerializer.valueOf("METADATA_0_0_2")
                def profile = ArtProfileKt.ArtProfile(file)
                def keys = new ArrayList(profile.profileData.keySet())
                def sortedData = new LinkedHashMap()
                Collections.sort keys, new DexFile.Companion()
                keys.each { key -> sortedData[key] = profile.profileData[key] }
                new FileOutputStream(file).with {
                    write(version.magicBytes$profgen)
                    write(version.versionBytes$profgen)
                    version.write$profgen(it, sortedData, "")
                }
            }
        }
    }
}

nielsvanvelzen · 2023-01-24T15:41:14Z

Our Gradle build files use Kotlin and it looks like the compiler is slightly more strict. We can't use the properties/functions on ArtProfileSerializer because their visibility is internal.

obfusk · 2023-01-24T18:23:16Z

We can't use the properties/functions on ArtProfileSerializer because their visibility is internal.

I was afraid it would not work with build.gradle.kts because of the visibility, but thanks for confirming 😅

You might be able to put it in a separate .gradle file (e.g. fix-profm.grade) and use apply(from = "fix-profm.gradle") from build.gradle.kts, but I haven't tested this.

obfusk · 2023-01-24T18:45:29Z

A quick test suggest it will fail to find the required imports via apply(from = ...); but perhaps you can make it work?

FWIW, using the python script to sort instead of com.android.tools.profgen should work with Kotlin.

obfusk · 2023-01-24T18:47:48Z

You can also wait for a new AGP version with the fix, but that might take a while.

obfusk · 2023-01-24T20:42:24Z

You can put this in app/fix-profm.gradle:

buildscript {
    repositories {
        google()
        mavenCentral()
    }
    dependencies {
        // NB: use this with gradle/libs.versions.toml (and modify as needed):
        classpath "com.android.tools.build:gradle:${libs.versions.android.gradle.get()}"
    }
}

import com.android.tools.profgen.ArtProfileKt
import com.android.tools.profgen.ArtProfileSerializer
import com.android.tools.profgen.DexFile

project.afterEvaluate {
    tasks.compileReleaseArtProfile.doLast {
        outputs.files.each { file ->
            if (file.toString().endsWith(".profm")) {
                println("Sorting ${file} ...")
                def version = ArtProfileSerializer.valueOf("METADATA_0_0_2")
                def profile = ArtProfileKt.ArtProfile(file)
                def keys = new ArrayList(profile.profileData.keySet())
                def sortedData = new LinkedHashMap()
                Collections.sort keys, new DexFile.Companion()
                keys.each { key -> sortedData[key] = profile.profileData[key] }
                new FileOutputStream(file).with {
                    write(version.magicBytes$profgen)
                    write(version.versionBytes$profgen)
                    version.write$profgen(it, sortedData, "")
                }
            }
        }
    }
}

And add this to app/build.gradle.kts:

apply(from = "fix-profm.gradle")

And it should work.

nielsvanvelzen · 2023-03-03T20:32:48Z

Thanks @obfusk, I've applied this workaround in #2558 and will backport it to our next patch release which will likely be on the 12th (edit: or earlier). @linsui do I need to make changes to f-droid data after the release or will you deal with that part?

linsui · 2023-03-04T03:53:40Z

You can open an MR or ping me when it's ready.

nielsvanvelzen · 2023-03-14T12:56:22Z

@linsui https://github.com/jellyfin/jellyfin-androidtv/releases/tag/v0.15.5 includes the workaround for reproducible builds. Let me know if this issue persists or something else comes up. Thanks for the report!

licaon-kter · 2023-03-14T14:18:55Z

A quick test did fail to Repro :(

nielsvanvelzen · 2023-03-14T18:29:33Z

Hmm weird, I tested it by creating multiple builds locally and using diff -r to see if there were binary differences and it worked after the pull request. I guess I'll need to do some more testing later. Was the issue still in the baseline files in your quick test?

licaon-kter · 2023-03-14T18:47:39Z

~~It's the "number of CPUs" issue, will test 1/2/4/etc cores and adapt the recipe.~~
see below

licaon-kter · 2023-03-15T09:20:55Z

Ok, builds fine, I've missed that it needs Java17 now 🥳

nielsvanvelzen · 2023-03-15T09:22:48Z

Awesome, thanks for checking! Glad to hear it's working now.

licaon-kter · 2023-03-15T09:26:39Z

Done https://gitlab.com/fdroid/fdroiddata/-/commit/260ab2dd85fe5739476f756b6233ae556d36344e

Thanks for sticking with it :)

linsui added the bug Something isn't working label Apr 21, 2022

nielsvanvelzen mentioned this issue Aug 4, 2022

Remove generation timestamp from AboutLibraries data file #1933

Merged

nielsvanvelzen closed this as completed in #1933 Aug 5, 2022

nielsvanvelzen reopened this Sep 16, 2022

nielsvanvelzen changed the title ~~F-Droid reproduciable build failed~~ F-Droid reproduceable build failed Feb 3, 2023

nielsvanvelzen mentioned this issue Mar 3, 2023

Add workaround for non reproducible baseline profiles in AGP #2558

Merged

nielsvanvelzen closed this as completed in #2558 Mar 5, 2023

nielsvanvelzen reopened this Mar 14, 2023

nielsvanvelzen closed this as completed Mar 15, 2023

nielsvanvelzen mentioned this issue May 17, 2023

Add workaround for non reproducible baseline profiles in AGP to libre builds jellyfin/jellyfin-android#1065

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

F-Droid reproduceable build failed #1639

F-Droid reproduceable build failed #1639

linsui commented Apr 21, 2022

sourpatched commented Apr 21, 2022

nielsvanvelzen commented May 7, 2022

linsui commented May 8, 2022

linsui commented May 31, 2022

licaon-kter commented Aug 3, 2022

nielsvanvelzen commented Aug 3, 2022

nielsvanvelzen commented Aug 5, 2022

licaon-kter commented Sep 15, 2022

linsui commented Jan 9, 2023

nielsvanvelzen commented Jan 9, 2023

obfusk commented Jan 10, 2023

linsui commented Jan 10, 2023

linsui commented Jan 10, 2023

linsui commented Jan 11, 2023 •

edited

Loading

nielsvanvelzen commented Jan 11, 2023 •

edited

Loading

linsui commented Jan 16, 2023

obfusk commented Jan 20, 2023

nielsvanvelzen commented Jan 24, 2023

obfusk commented Jan 24, 2023 •

edited

Loading

obfusk commented Jan 24, 2023

obfusk commented Jan 24, 2023

obfusk commented Jan 24, 2023

nielsvanvelzen commented Mar 3, 2023 •

edited

Loading

linsui commented Mar 4, 2023

nielsvanvelzen commented Mar 14, 2023

licaon-kter commented Mar 14, 2023

nielsvanvelzen commented Mar 14, 2023

licaon-kter commented Mar 14, 2023 •

edited

Loading

licaon-kter commented Mar 15, 2023

nielsvanvelzen commented Mar 15, 2023

licaon-kter commented Mar 15, 2023

F-Droid reproduceable build failed #1639

F-Droid reproduceable build failed #1639

Comments

linsui commented Apr 21, 2022

Describe the bug

Logs

Application version

Where did you install the app from?

Device information

Android version

Jellyfin server version

sourpatched commented Apr 21, 2022

nielsvanvelzen commented May 7, 2022

linsui commented May 8, 2022

linsui commented May 31, 2022

licaon-kter commented Aug 3, 2022

nielsvanvelzen commented Aug 3, 2022

nielsvanvelzen commented Aug 5, 2022

licaon-kter commented Sep 15, 2022

linsui commented Jan 9, 2023

nielsvanvelzen commented Jan 9, 2023

obfusk commented Jan 10, 2023

linsui commented Jan 10, 2023

linsui commented Jan 10, 2023

linsui commented Jan 11, 2023 • edited Loading

nielsvanvelzen commented Jan 11, 2023 • edited Loading

linsui commented Jan 16, 2023

obfusk commented Jan 20, 2023

nielsvanvelzen commented Jan 24, 2023

obfusk commented Jan 24, 2023 • edited Loading

obfusk commented Jan 24, 2023

obfusk commented Jan 24, 2023

obfusk commented Jan 24, 2023

nielsvanvelzen commented Mar 3, 2023 • edited Loading

linsui commented Mar 4, 2023

nielsvanvelzen commented Mar 14, 2023

licaon-kter commented Mar 14, 2023

nielsvanvelzen commented Mar 14, 2023

licaon-kter commented Mar 14, 2023 • edited Loading

licaon-kter commented Mar 15, 2023

nielsvanvelzen commented Mar 15, 2023

licaon-kter commented Mar 15, 2023

linsui commented Jan 11, 2023 •

edited

Loading

nielsvanvelzen commented Jan 11, 2023 •

edited

Loading

obfusk commented Jan 24, 2023 •

edited

Loading

nielsvanvelzen commented Mar 3, 2023 •

edited

Loading

licaon-kter commented Mar 14, 2023 •

edited

Loading