Enable retry on network failures

[arjunsuresh]

We often see CM script runs failing due to netwok failures like this. It'll be good to add a retry mechanism for such failures to improve the user experience and reduce the failures of automatic runs.

[anandhu-eng]

Hi @arjunsuresh , this would be useful. Should this be kept on by default or should it be controlled through any env variable? I'm wondering if there is a case where user wants to turn it off

[arjunsuresh]

@anandhu-eng I think we should enable it by default and let users an ENV variable to turn it off for any reason. But first we need to list out the places where we need this. Below are some of them. We should probably try it on one, and if it works as expected move to the remaining places.

1. git clone
2. System util installation
3. pip package installation

[arjunsuresh]

The below failure is seen many times in our github actions. Trying the fix --dns 8.8.8.8 --dns 8.8.4.4 to docker run command.

2024-11-05T13:38:34.8624636Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/libsystemd0_245.4-4ubuntu3.24_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8629875Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/a/argon2/libargon2-1_0~20171227-0.2_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8634850Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/l/lvm2/libdevmapper1.02.1_1.02.167-1ubuntu1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8639786Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/i/iptables/libip4tc2_1.8.4-3ubuntu2.1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8642386Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/k/kmod/libkmod2_27-1ubuntu2.1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8644257Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/systemd-timesyncd_245.4-4ubuntu3.24_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8646140Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/systemd_245.4-4ubuntu3.24_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8648397Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/l/lvm2/dmsetup_1.02.167-1ubuntu1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8650427Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/g/gobject-introspection/libgirepository-1.0-1_1.64.1-1~ubuntu20.04.1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8652805Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/g/gobject-introspection/gir1.2-glib-2.0_1.64.1-1~ubuntu20.04.1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8654823Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/d/dbus-python/python3-dbus_1.2.16-1build1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8656630Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/pygobject/python3-gi_3.36.0-1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8658027Z E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

[arjunsuresh]

git clone failures

Cloning into 'inference'...
error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8)
error: 8093 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output