Benchmarking Cachix vs Attic

[dsalaza4]

Hey,

I've been playing with Attic during the last couple weeks and got to a point where I wanted to compare how fast it was in terms of building already-cached derivations compared to Cachix.

My main focus was finding a way to build derivations in the least amount of time.

Currently, my company uses Cachix for AWS EC2 machines hosted on us-east-1. Cachix hosts its servers and buckets in the same region and does not allow for much tweaking, meaning that we're probably getting the best performance out of it right now.

This is the most basic schema I can think for describing it:

AWS Machine ---run-nixos-container--> Container ---pull-cache--> Cachix

I figured maybe hosting a local attic cache within the AWS machine and then accessing it from the container via localhost would be much faster, as data, although still using http, would never leave the machine but rather only be sent via localhost.

AWS Machine ---host-attic--> 127.0.0.1
AWS Machine ---run-nixos-container--> Container ---pull-cache--> 127.0.0.1

Here are some results I got:

1. With many small derivations: https://gitlab.com/fluidattacks/universe/-/issues/12386#note_2000625060
2. With one big derivation: https://gitlab.com/fluidattacks/universe/-/issues/12386#note_2000642907

Please note I did not build these benchmarks with the purpose of other people being able to reproduce them, but rather as a simple way of verifying if I was actually seeing an increase in build speed when using attic locally compared to cachix via internet.

The results are pretty confusing to me, as even though the data remains in the machine and travels via localhost, I'm actually seeing a decrease in performance for most scenarios.

With this in mind, I have several questions I hope someone would shine some light upon:

1. What is the recommended server configuration for prioritizing cache download/decompression in order to build derivations as fast as possible?
2. Are there any bottlenecks I may not be considering with this architecture?
3. Are there any other ways for making pulling cache faster, like completely getting rid off http and using disk files instead (like using a second /nix/store)?

[dsalaza4]

I decided to test hosting the attic server on a separate machine and communicate the machine building with the cache server through the AWS VPC.

Results show a 30% build speed improvement.

https://gitlab.com/fluidattacks/universe/-/issues/12386#note_2002396581

This leads me to think that the bottleneck when hosting the server within the same machine that builds is most likely CPU.