Our recent works on making copr sign process more effective

[pkking]

Hi team, im very excited to share our recent works on copr and looking forward to hear you opinions :)

Why we need a faster sign process?

Once upon a while we have to sign a lot of packages in our copr instances(called EUR) because we need to switch to a new signing key.
When i courageously run the copr_fix_gpg.py for my first 2000 packages, the whole process halt for about 45mins and meanwhile i am so worried it will jam forever when i then start to sign the left 20,000 packages.

After this heartquake signing process, i take some time to investigate the copr sign process.

How copr sign a rpm

Here's a image illustrating how copr sign a rpm:

On the client-side(copr-backend):

1. backend called a RESTful API /gen_key to gen the key when new project is created/forked.
2. backend called the sign tool in obs-signd rpm package to sign each rpm separately.

On the server-side(copr-keygen):

1. once the /gen_key API is called, copr-keygen will call pgp2 to generate a keypair and store it in the database.
2. when backend calls sign tool to sign a rpm, the tool will establish a TCP connection with obs-signd running on keygen node and send the rpm payload to it.

Why backend sign not so fast

1. backend call sign to sign rpm one by one, actually each sign process were irrelevant
2. sign can't resign a rpm, so we must unsign && sign the rpm each time
3. obs-signd also do not support concurrent

Way to speed up

At the early of 2023, openEuler community open sourced signatrust as a highly secure, async and efficient solution for signing Linux packages and binaries platform.

Signatrust itself provides secure and high throughput for all signing requests, and that's what we thought a mordern signing infrastructure should be. Here

Why signatrust is faster

1. the sign client start multi workers to sign rpms parallelly
2. signatrust provided a more quickly API to check key existence
3. signatrust will construct the rpm with signature in place, so we do not need unsigned the rpm before signing
4. each sign request will be handled in async way, so multiple requests can be processed concurrently

Benchmarks

Here are some benchmarks comparing the performance of signatrust vs the obs-signd approach:

Lets explaine the benchmarks:

1. both obs-signd and signatrust runs on a 8u8G kubernetes pod and within same namespace with copr-backend
2. we tried three kind of workload to simulate most our daily works on copr:

• sign few package: most of the build runs workload like this
• sign some packages: most of the project forked or batch rebuild
• sign massive packages: some rare but important SOP/maintaince case

The bench results showed signatrust were 3x-10x faster than obs-signd in all three cases. This is mainly because:

1. signatrust client used multi-threading to sign rpms in parallel, thats why in few package case it was 3x faster
2. signatrust will not unsign rpm before signing, this avoids unnecessary rpm repackaging
3. signatrust server using async io model to handle multiple signing requests concurrently

And also, we monitord obs-signd CPU and Memory usage, it's always very low during signing process. This indicates obs-signd is not fully utilizing the server resources and not handling requests concurrently.

What now

openEuler copr instance EUR(openEuler User Repo) have already finished migrating our sign backend service from obs-signd to signatrust

To sum up in a word: each time the user build a single package on a chroot cost less 5~10 seconds than before

see the i3 builds using obs-signd vs using signatrust

Still we patched backend to make sign backend plugable, if you were interested, i'd like make a PR :)

[praiskup]

Thank you for the analysis, @pkking!

One missing point we should consider is signing of large (S)RPMs. Consider that the RPMs are often larger than a few MBs, and sometimes are also very large >= 1GB (we have an internal use-case where RPMs are >= 3GB). The obs-sign software is very efficient here WRT the client-server approach (the client calculates the hashing part of the signature, so the whole large RPM blob doesn't have to be sent to the server for actual signing). Have you tried to measure this use-case and compare?

Still we patched backend to make sign backend plugable, if you were interested, i'd like make a PR :)

Definitely. As long as it is opt-in, it's definitely OK to have the support upstream!

[pkking]

Oh, this case is not covered by our test cases, will try to test and replay later :)

[TommyLike]

@praiskup currently we only calculate the rpm digest header at the client side, please check this , the payload and header will be transferred to the server side cause it will be used for the PGP library directly, not sure about the obs-sign detail although, considering the signing system and COPR will be deployed in the same network area, I am wondering what is the possibility of having rpm larger than 1GB in the COPR environment and what exactly the impact of these, we will try to find this, thanks.

[Conan-Kudo]

There are several such examples in Fedora. chromium, texlive, and 0ad-data are good test cases.

[TommyLike]

@Conan-Kudo Thanks we will test these.

[TommyLike]

@praiskup there is another point, considering the difference of rpm signature v3 and v4 there would be big performane improvement if no payload is transfered on the network, so, what's the strategy for COPR to stop support signature v3? check this: rpm-software-management/rpm#863

[praiskup]

@pkking WRT the mass-resign process, note that we plan to use PULP as the RPM repository manager in the future. So you might want to work on integrating Signatrust with PULP, too?

[TommyLike]

@praiskup Just checked the document of pulp and pulp rpm, I think it does not support signing the rpm file but only the repo metadata file currently?

[ipanova]

That's correct, only metadata files for now.

[pkking]

@pkking WRT the mass-resign process, note that we plan to use PULP as the RPM repository manager in the future. So you might want to work on integrating Signatrust with PULP, too?

Its my fisrt glance on PULP, it seems to be a funny project who also happened to solv many of our problems and i will talk to signatrust team to discuss the integration :)

[Conan-Kudo]

cc: @bmbouter and @dralley for this discussion. They can provide useful information about Pulp. :)

[ipanova]

@pkking at the moment we have captured in a ticket native obs-signd support for signing Pulp software, we are opened to look into Signatrust if @praiskup and rest think COPR will benefit from it. A pulp ticket with detailed info is more than welcome and as always if you feel like you have the desire and motivation to contribute we will be more than happy to assist you through questions.

[praiskup]

I haven't done an analysis myself, so I trust @pkking that it performs nicely and is worth it (even if only their instance used it). I just miss the analysis of signing rather big RPMs (see above).

Also, once Copr is on PULP and RPMs are stored somewhere in s3, re-signing effectively (what @pkking's team seems to do heavily) will be an interesting problem to solve even with obs-sign.

[dralley]

Yes, realistically I don't see that being especially practical with header-and-payload signatures. But even with header-only signatures, we'd still need to persist a checksum of the header which we don't currently do, OR download the header bytes using a range request for processing.

[TommyLike]

@praiskup @Conan-Kudo

We conducted performance tests in two scenarios:

Scenario 1: Common scenarios for RPM package signing. The test files range from a few MB to several GB. There are 4,000 RPM packages occupying 19 GB of disk space in total. All RPM files can be downloaded at: https://repo.openeuler.org/openEuler-22.03-LTS/source/Packages/

Scenario 2: Extreme scenario where all RPM packages are large files. Actually each RPM has a size of 1.8 GB, with a total of 50 files, occupying 100 GB of disk space. The test RPM package can be downloaded at: https://dl.fedoraproject.org/pub/fedora/linux/releases/38/Everything/source/tree/Packages/s/speed-dreams-2.2.3-6.fc38.src.rpm

Test Environment:
Each test consists of one client and one server, with each instance restricted to 8 CPU cores and 8 GB of RAM. The client runs with a concurrency of 8. Note that the obs-sign client does not support simultaneous signing. To address this, we used a script to simultaneously launch 8 obs-sign clients for signing.

Test Results:
obs-sign(v3 signature):
Scenario 1: 20 minutes and 51 seconds
Scenario 2: 18 minutes and 5 seconds

signatrust(v3 signature):
Scenario 1: 4 minute and 43 seconds
Scenario 2: OOM

signatrust(v4 signature):
Scenario 1: 1 minute and 53 seconds
Scenario 2: 18 minutes and 34 seconds

CPU and Memory Consumption for Scenario 2
obs-sign:
Client CPU:

Client Memory:

Server CPU:

Server Memory:

signatrust:
Client CPU:

Client Memory:

Server CPU:

Server Memory:

Conclusion:
In common scenarios, Signatrust significantly improves the efficiency of RPM package signing by taking advantage of asynchronous processing.
In the extreme scenario of large RPM files. obs-sign, due to its approach of only transmitting digests for signature computation, it can reduce network consumption and improve the efficiency for RPM v3 signature, but when compared to the RPM V4 Signature mode that only signs the header, the overall efficiency is similar.

[praiskup]

Thank you for taking a look. I miss the obs-sign(v4 signature) part, don't you have results also from there? I suppose that Signatrust is much faster with v4 signatures? We have the #2757 btw, which seems to be caused by -4 option, right?

[TommyLike]

if obs-sign will send digest for v3 signature, I guess the v4 signature will be a little(not too much) faster because the calclucation is not needed, but we will have a try. @praiskup

[dralley]

It's maybe best not to talk about it in terms of v3 signatures and v4 signatures since that overlaps with PGP v3 and v4, which is actually entirely unrelated.

There are header-only signatures, and header-and-payload signatures, where the latter is also sometimes known as RPMv3 signatures.

On recent distros, RPM does not produce header-and-payload / RPMv3 signatures by default rpm-software-management/rpm#1093

Thus header-and-payload signatures ought to cease being particularly relevant in a few months when EL7 goes end-of-life. I assume at that point COPR would start dropping EL7 support?

[TommyLike]

thanks for your explaination. @dralley

[dralley]

Which I suppose you mentioned here already #2935 (reply in thread)