Try rsync instead of archive + scp

[artem-zinnatullin]

Though in our case we generate about 10k build files on each build and rsync might be slower than simple tar + scp.

See this discussion for more info: https://twitter.com/steventannz/status/811460935101493248

[steventannz]

Ah, I see the difference now. I've omitted syncing some of the build files, namely the intermediates as they took extra time to sync. They're probably still needed for unit tests to integrate nicely with the IDE but it's possible to just run them on the remote server instead.

For the typical use case of compiling and deploying an APK, those files are not needed and a lot of time can be saved by excluding them. Here's a snippet with project details omitted of the script that I'm using:

echo Syncing to remote machine
rsync -az --delete \
--exclude='**/.git/' \
--exclude='**/local.properties' \
--exclude='**/build/intermediates' \
--exclude='**/build/tmp' \
--exclude='.idea' \
-e "ssh" ./ {SERVER}:/tmp/android/$(whoami)

echo Building on remote machine
ssh {SERVER} "cd /tmp/android/$(whoami)/{PROJECT}; ./gradlew --daemon --parallel assembleDebug"

echo Syncing back to local machine
rsync --delete -avz --partial \
--exclude='**/.git/' \
--exclude='**/local.properties' \
--exclude='**/build/intermediates' \
--exclude='**/build/tmp' \
--exclude='.idea' \
-e "ssh" {SERVER}:/tmp/android/$(whoami)/ ./

This approach will only sync changed files and will be beneficial to projects that use a lot of assets that do not change often between compilation.

[artem-zinnatullin]

Without intermediates IDE won't see new build files (classes, etc) which will force you to do local Gradle sync pretty often :(

[steventannz]

Hmm with this script, I haven't been prompted for any syncs. Mainly only getting back the compiled APKs and a few generated files which should not trigger a sync requirement.

[artem-zinnatullin]

For example: we use annotation processing that generates classes, without pulling intermediates we won't see this generated classes in the IDE and that'll break a lot of code editing for us.

[gabrielittner]

Generated code goes into build/generated and when the build cache is enabled exploded aar's live there and not in intermediates. So maybe it's enough to sync generated and outputs when build cache is enabled and otherwise also sync intermediates/exploded-aar?

[artem-zinnatullin]

@gabrielittner right right, but to launch tests we need intermediate/classes.

Anyway, this will be solved by introdusing .mainframerignore and .mainframerinclude (or some other way to pass what to include and what to ignore) as part of #19!

[ligol]

The discussion only talk about the build file and pulling these file from the server, but what about the uploading part of the source code. I think on this side rsync could be better than tar+scp, especially for a project with a lot of res file like drawable.

We could maybe use both method.

What do you think?

[steventannz]

I included build/intermediates again and ran a couple of quick tests between tar+scp and rsync on my own project (excluding the time it took building it). This was done over a 1Gbps ethernet connection.

tar + scp

Time to local tar excl */build:

time tar c --exclude='local.properties' --exclude='.gradle' --exclude='.idea' --exclude='.git' --exclude='artifacts' --exclude='captures' --exclude='build' --exclude='*/build' -f archive.tar {PROJECT_FOLDER}/

real	0m0.692s
user	0m0.057s
sys	0m0.236s

Time to scp from local to remote:

time scp archive.tar {BUILD_SERVER}:/tmp/android/$(whoami)/archive.tar
archive.tar                                                                 100%   78MB  78.4MB/s   00:00    

real	0m1.001s
user	0m0.470s
sys	0m0.386s

Time to remote untar:

time tar -xf archive.tar

real	0m0.429s
user	0m0.034s
sys	0m0.357s

Time to remote tar just /build and */build:

time tar \
> -c \
> --exclude='kotlin' \
> --exclude='tmp' \
> -f archive.tar \
> build/ */build

real	0m4.446s
user	0m0.261s
sys	0m1.275s

13,959 items

Time to scp from remote to local:

time scp archive.tar.gz {BUILD_SERVER}:/tmp/android/$(whoami)/archive.tar
archive.tar                                100%  273MB 136.5MB/s   00:02    

real	0m2.906s
user	0m1.485s
sys	0m1.363s

368.5MB

Time to local untar:

time tar -xf archive.tar

real	0m2.621s
user	0m0.155s
sys	0m2.151s

Total time: 12.905 seconds

rsync

First pass from clean build

Time to rsync to remote:

rsync -a --delete \
--exclude='**/.git/' \
--exclude='**/local.properties' \
--exclude='.idea' \
-e "ssh" ./ {BUILD_SERVER}:/tmp/android/$(whoami)

real	0m6.946s
user	0m2.996s
sys	0m3.246s

Time to rsync back to local:

rsync --delete -av --partial \
--exclude='**/.git/' \
--exclude='**/local.properties' \
--exclude='.idea' \
-e "ssh" {BUILD_SERVER}:/tmp/android/$(whoami)/ ./

sent 381158 bytes  received 133716752 bytes  14115569.47 bytes/sec
total size is 473351233  speedup is 3.53
real	0m9.475s
user	0m2.831s
sys	0m5.976s

8638 changed files synced

Total time: 16.421 seconds

Second pass with 3 lines of edited code

Time to sync to remote:

rsync -a --delete \
--exclude='**/.git/' \
--exclude='**/local.properties' \
--exclude='.idea' \
-e "ssh" ./ {BUILD_SERVER}:/tmp/android/$(whoami)

real	0m0.677s
user	0m0.145s
sys	0m0.163s

Time to sync back to local:

rsync --delete -av --partial \
--exclude='**/.git/' \
--exclude='**/local.properties' \
--exclude='.idea' \
-e "ssh" {BUILD_SERVER}:/tmp/android/$(whoami)/ ./

sent 301456 bytes  received 6170583 bytes  1438230.89 bytes/sec
total size is 473351256  speedup is 73.14

real	0m4.110s
user	0m1.314s
sys	0m1.706s

2436 changed files synced

Total time: 4.787 seconds

Results

From my quick tests, tar + scp takes roughly the same amount of time every build.

rsync is slower than tar + scp by 3.516 seconds on its first run (16.421 vs 12.905) as it has to copy the entire project directly at first and also sync the entire build folder after it has finished building on remote.

However, there are far fewer files (8638 build files generated at first, 2436 changed on next run) on the second run to rsync to remote and back. This is significantly faster than tar + scp for every subsequent run (4.787 vs 12.905) as most of the files in build/ did not change and majority of the core project is already on remote and does not need to be re-copied each time.

[arturdryomov]

@steventannz Huh, this is very interesting, thank you for the stats! We’ll try it out locally in different configurations to be sure we want to switch, but this is definitely looks promising!

[artem-zinnatullin]

@steventannz good results, thanks! If you have time — feel free to make PR with switching to rsync so we could try it in our setup, if no — we'll do it ourselves and if it'll work better we'll open a PR :)

[kokeroulis]

Today i was experimenting with rsync, and i can verify the above. Only the first time rsync is slower, but after that, its way faster.

I have forked this project in order to experiment with the rsync. With the feedback from this issue, i have added the rsync functionality.

The code is not very good (this is the first time that i am writing bash), but i thought that it might help

https://github.com/kokeroulis/mainframer

I could create a PR, but it might take a few days to fix the code because of christmas.
So if you need this assap you are free to reuse it or whatever :)

[artem-zinnatullin]

@kokeroulis thanks! We'll try it.

@ming13 let's dive into man rsync and google for possible issues that rsync may cause in our use case, I also think we'll need to try to live it for a week or so before releasing it in publically available version of mainframer.

[artem-zinnatullin]

Closed by #35!