Optimize pyinstaller build

[dromer]

As discussed here: plugdata-team/plugdata#1453

Execution of the pyinstaller version of hvcc is very slow. Next to the fact that executing from a JUCE subprocess also seems to give some performance penalty.

Converting a fairly complex patch (will link something shortly) gives the following compile times:

method	time
local hvcc	949.65ms
plugdata toolchain Heavy	33097.69ms
manual toolchain Heavy	12091.99ms
local Heavy	1312.25ms

The final result was a manual pyinstaller build of latest hvcc and is much more in line with what is expected.
It seems our build in the toolchain is particularly sluggish by over a factor difference.

It would be good to investigate how to make sure that the toolchain build gets the same performance as local pyinstaller builds.

[timothyschoen]

Interesting benchmark! That does look bad...

Any idea what I'm doing differently with my pyinstaller build?

[dromer]

Hmmm, for the Linux version this might be in part because it's using debian:bullseye which uses python3.9

I'm now trying to test automated builds using debian:bookworm and python3.11 - see if that makes a difference (my local system is bookworm with 3.11 as well)

[dromer]

Ok, this gives similar result as the local pyinstaller build: Total compile time: 1297.48ms
So hopefully this is all we need to get some better performance!

Still likely that running it from a subprocess will be much slower, but then it's ~3 seconds instead of ~30 seconds which is absolutely much better :)

I'll build the full toolchain and report back soon :)

[timothyschoen]

Sweet! One thing to keep in mind, is that that will make it link against a libc that was packaged with a distro from 2023. That's a bit recent, and could lead to compatibility issues with older distros.

Might be better to (somehow) install python 3.11 on a debian bullseye runner...

[dromer]

yeah perhaps, or we can use ubuntu-20.04 with python3.11 instead of the current privileged container runtime

[timothyschoen]

yeah perhaps, or we can use ubuntu-20.04 with python3.11 instead of the current privileged container runtime

oh yeah that'd make sense

[dromer]

Hmm, oddly enough running from plugdata it takes much longer: Total compile time: 19103.04ms
VS running the same Heavy binary locally: Total compile time: 1297.15ms

That's nearly 15 times slower! o.O

[timothyschoen]

Damn, maybe we're reading the process output too often or something?

[timothyschoen]

At line 83 of ExportingProgressView.h, you see "Time::waitForMillisecondCounter(Time::getMillisecondCounter() + 100);"

You could try to make the wait much longer, see if that improves performance?

[timothyschoen]

Alright, just updated the build server to run on Ubuntu 20.04 (oldest I could find with Python 3.11 available) with the newer python. So that should at least help a bit.

[dromer]

I think by default 20.04 uses python3.10 no?

But anyway it doesn't seem that this is our biggest bottleneck. I'll try patching ExportProgressView.h and see what that does ..

[edit: I made the wait + 1000 and it's still Total compile time: 19554.73ms]

[timothyschoen]

According to the ubuntu repo site, 22.04 comes with 3.10, 20.04 comes with 3.8.2. So we needed to install python from an external repo anyway, might as well do it on the oldest distro.

Ah, unfortunate that that wasn't it. Is this the time in ms that heavy reports? Here on macOS it's "133.29ms" for an empty patch, is that close to what you get?

[dromer]

Yeah an empty patch is Total compile time: 100.98ms so comparable.
Maybe it has something to do with disk access to parse all the included library files?

[timothyschoen]

Though if we're only getting around 100ms for an empty patch, but 19554ms for a large patch, that would suggest that this is not the overhead of starting a process. So maybe it is something about the file access? But wouldn't that be the same with regular Heavy?

[timothyschoen]

Oh wait, IIRC it unzips itself and all of its resources at startup right? That would make it take longer. But I'm not sure how that delay could be detected from inside Heavy?

I think it's also an option to not make a "zipped" package with pyinstaller. Maybe that is what you are doing with your local pyinstaller builds?

[timothyschoen]

This is the "--windowed" flag, and I've just pushed a commit that disabled it. Seems to make a bit of a difference here at least, an empty patch now takes about 80ms instead of 130ms.

[dromer]

My local pyinstaller build uses the same commands, but the differences I am now reporting is the same Heavy build from the toolchain, just local VS plugdata execution and that is 15x slower when executed from plugdata.

These timings are indeed Heavy timing its own endtime - starttime.
I still find this very strange ..

[dromer]

The latest build with 20.04 default python takes:

• local: Total compile time: 1630.67ms
• plugdata: Total compile time: 20133.07ms

Local is about 30% slower than the bookworm/py311 build, but plugdata is ~comparable as before. This leads me to believe that the main bottleneck is still in plugdata 🤔

[dromer]

We're using a newer pyinstaller that has everything self-contained. It will take a bit longer to start (decompressing to /tmp), but the actual compile times will need to be checked.

[timothyschoen]

We're using a newer pyinstaller that has everything self-contained. It will take a bit longer to start (decompressing to /tmp), but the actual compile times will need to be checked.

Heh, we already used to do that, but I reverted it because I suspected it made stuff slower. But if you don't notice a difference, I guess either is fine

[dromer]

It only slows down the startup of the program, by like a few 100 ms maybe.
The actual compile time execution is what I'm still worried about.

We are still using py311 and I'm suspecting that py312 will be faster .. so hopefully we can move to that for the next release ..

[dromer]

Hmm, trying the latest toolchain on my laptop and there it gives (with this "big" patch that I've been trying with): Total compile time: 2422.83ms

Whereas on my workstation (that is much more powerful) I still get something like ~20s - maybe there is something strange going on with the builds on my workstation. I will try to build a completely fresh plugdata there later this week.

[timothyschoen]

I think the reason it's so slow on macOS is that I'm currently not signing and notarising the Heavy toolchain. I've seen the same thing happen with externals on pd-vanilla: if you download binaries from within a codesigned application (like how plugdata downloads the toolchain), they will work normally but take a lot longer to start up.

[dromer]

I didn't know it was also slow on macOS?

We have a bit of startup time with the bundled pyinstaller build of course, but the numbers I've been sharing are execution time of converting a patch measured from inside the program.

[timothyschoen]

Yeah, basically compilation is fast but it takes a while to get started. I noticed this before, but could never explain it before

[dromer]

Hmm, I see. Maybe we can already show something in the console when we know the Heavy command is running. Right now we only get output once it is done.

At least this will give the illusion of progress :P
(and you can then better measure the impact of this startup-time)

[dromer]

Hmm, a different complex patch (one I ran on the Field at DMF) takes 17307.16ms on my desktop and 2685.44ms on my laptop .. so definitely need to figure out why my desktop is so slow o.O