Flaky continuous integration

[AndreMiras]

Abstract

Ticket to keep a track and find possible solutions for flaky tests we saw on Travis.

Known errors

curl: (56) GnuTLS recv error

Happens during CrystaX NDK download, the error is:

curl: (56) GnuTLS recv error (-9): A TLS packet with unexpected length was received.

Example log: https://travis-ci.org/kivy/python-for-android/jobs/397824994
Possible solution, do some download retry:
https://stackoverflow.com/questions/10568497/how-does-curl-retry-max-time-seconds-work

curl: (35) gnutls_handshake() failed

Also happens during CrystaX NDK download, error:

curl: (35) gnutls_handshake() failed: Error in the pull function.

Example logs:

• https://travis-ci.org/kivy/python-for-android/jobs/381502437
• https://travis-ci.org/kivy/python-for-android/jobs/381297644
• https://travis-ci.org/kivy/python-for-android/jobs/381499077

Possible solution also some retries.

curl: (7) Failed to connect to eu.crystax.net port 443: Connection timed out

Again during CrystaX NDK download, error:

curl: (7) Failed to connect to eu.crystax.net port 443: Connection timed out
The command '/bin/sh -c curl --location --progress-bar "${CRYSTAX_NDK_DL_URL}" --output "${CRYSTAX_NDK_ARCHIVE}" --insecure &&     bsdtar -xf "${CRYSTAX_NDK_ARCHIVE}" --directory "${ANDROID_HOME}"     --exclude=crystax-ndk-${CRYSTAX_NDK_VERSION}/docs     --exclude=crystax-ndk-${CRYSTAX_NDK_VERSION}/samples     --exclude=crystax-ndk-${CRYSTAX_NDK_VERSION}/tests     --exclude=crystax-ndk-${CRYSTAX_NDK_VERSION}/toolchains/renderscript     --exclude=crystax-ndk-${CRYSTAX_NDK_VERSION}/toolchains/x86_64-*     --exclude=crystax-ndk-${CRYSTAX_NDK_VERSION}/toolchains/llvm-*     --exclude=crystax-ndk-${CRYSTAX_NDK_VERSION}/toolchains/aarch64-*     --exclude=crystax-ndk-${CRYSTAX_NDK_VERSION}/toolchains/mips64el-* && 	ln -sfn "${CRYSTAX_NDK_HOME_V}" "${CRYSTAX_NDK_HOME}" &&     rm -rf "${CRYSTAX_NDK_ARCHIVE}"' returned a non-zero code: 7

Example log: https://travis-ci.org/kivy/python-for-android/jobs/415163201 (rebuilt since then).

TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'

Happens when PyPI or whichever index server is down.
Error log: https://travis-ci.org/kivy/python-for-android/builds/375908909
Possible solution, retry or fallback on some mirrors.

error occurred while preparing SDK package Android SDK

Happens during SDK update using sdkmanager, error is:

Warning: An error occurred while preparing SDK package Android SDK Platform 19: Connection timed out (Connection timed out).

Example log: https://travis-ci.org/kivy/python-for-android/jobs/389474611
Implementing a retrying procedure may also help here.

UnknownHostException: services.gradle.org

Exception in thread "main" java.net.UnknownHostException: services.gradle.org

Happened during gradle (https://services.gradle.org/distributions/gradle-3.3-all.zip) download.
See full log attached:
gradle-download-error-log.txt

E: Failed to fetch

This one is more rare, but happened recently in the before_install section while running the apt update, see log extract below:

$ sudo apt update -qq
W: http://ppa.launchpad.net/couchdb/stable/ubuntu/dists/trusty/Release.gpg: Signature by key 15866BAFD9BCC4F3C1E0DFC7D69548E1C17EAB57 uses weak digest algorithm (SHA1)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packagecloud.io/github/git-lfs/ubuntu trusty InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6B05F25D762E3157
W: Failed to fetch https://packagecloud.io/github/git-lfs/ubuntu/dists/trusty/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6B05F25D762E3157
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/trusty-security/main/source/Sources.gz  Hash Sum mismatch
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/trusty-security/main/binary-amd64/Packages.gz  Hash Sum mismatch
W: Some index files failed to download. They have been ignored, or old ones used instead.
The command "sudo apt update -qq" failed and exited with 100 during 

See complete log: https://travis-ci.org/AndreMiras/python-for-android/jobs/486983237
Using travis_retry would probably be an easy workaround for that one.

Android SDK Platform 19: archive is not a ZIP archive.

During the docker build:

Step 21/36 : RUN "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-19" &&     "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-27" &&     "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "build-tools;${ANDROID_SDK_BUILD_TOOLS_VERSION}" &&     chmod +x "${ANDROID_SDK_HOME}/tools/bin/avdmanager"
 ---> Running in d32f2dfeef3f
Warning: An error occurred while preparing SDK package Android SDK Platform 19: archive is not a ZIP archive.
The command '/bin/sh -c "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-19" &&     "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-27" &&     "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "build-tools;${ANDROID_SDK_BUILD_TOOLS_VERSION}" &&     chmod +x "${ANDROID_SDK_HOME}/tools/bin/avdmanager"' returned a non-zero code: 1
The command "docker build --tag=p4a --file Dockerfile.py3 ." exited with 1.

Remark / current conclusions

So far the weakest point seems to be the NDK download, so that may probably be the first one we want to tacle.

Possible workaround / solutions

• kadwanev/retry
• travis_retry
• https://stackoverflow.com/a/8351489/185510

Edit

Recently the CrystaX NDK was dropped in profit for pure Python builds. However this ticket is still relevant for other hiccup. I'll update later with new errors we see the most often.

[inclement]

I just got the following issue on #957.2:

fter this operation, 469 MB of additional disk space will be used.
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/a/avahi/libavahi-common-data_0.7-3.1ubuntu1.2_amd64.deb  404  Not Found [IP: 91.189.88.149 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/a/avahi/libavahi-common3_0.7-3.1ubuntu1.2_amd64.deb  404  Not Found [IP: 91.189.88.149 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/a/avahi/libavahi-client3_0.7-3.1ubuntu1.2_amd64.deb  404  Not Found [IP: 91.189.88.149 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c apt -y update -qq     && apt -y install -qq --no-install-recommends openjdk-8-jdk     && apt -y autoremove     && apt -y clean' returned a non-zero code: 100
The command "docker build --tag=p4a --file Dockerfile.py3 ." exited with 100.
0.45s$ docker run p4a /bin/sh -c "$COMMAND"
Unable to find image 'p4a:latest' locally
docker: Error response from daemon: pull access denied for p4a, repository does not exist or may require 'docker login'.
See 'docker run --help'.
The command "docker run p4a /bin/sh -c "$COMMAND"" exited with 125.
Done. Your build exited with 1.

[inclement]

And on #957.1:

$ sudo apt update -qq
W: http://ppa.launchpad.net/couchdb/stable/ubuntu/dists/trusty/Release.gpg: Signature by key 15866BAFD9BCC4F3C1E0DFC7D69548E1C17EAB57 uses weak digest algorithm (SHA1)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packagecloud.io/github/git-lfs/ubuntu trusty InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6B05F25D762E3157
W: Failed to fetch https://packagecloud.io/github/git-lfs/ubuntu/dists/trusty/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6B05F25D762E3157
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/trusty-security/main/source/Sources.gz  Hash Sum mismatch
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/trusty-security/main/binary-amd64/Packages.gz  Hash Sum mismatch
W: Some index files failed to download. They have been ignored, or old ones used instead.
The command "sudo apt update -qq" failed and exited with 100 during .

[AndreMiras]

0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
 ---> 969a3f4a26f1
Removing intermediate container 57d25cfd4596
Step 20/36 : RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "build-tools;${ANDROID_SDK_BUILD_TOOLS_VERSION}" > /dev/null
 ---> Running in d57538dc1c28
 ---> 6d4614a46b1d
Removing intermediate container d57538dc1c28
Step 21/36 : RUN "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-19" &&     "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-27" &&     "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "build-tools;${ANDROID_SDK_BUILD_TOOLS_VERSION}" &&     chmod +x "${ANDROID_SDK_HOME}/tools/bin/avdmanager"
 ---> Running in d32f2dfeef3f
Warning: An error occurred while preparing SDK package Android SDK Platform 19: archive is not a ZIP archive.
The command '/bin/sh -c "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-19" &&     "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-27" &&     "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "build-tools;${ANDROID_SDK_BUILD_TOOLS_VERSION}" &&     chmod +x "${ANDROID_SDK_HOME}/tools/bin/avdmanager"' returned a non-zero code: 1
The command "docker build --tag=p4a --file Dockerfile.py3 ." exited with 1.
0.47s$ docker run p4a /bin/sh -c "$COMMAND"
Unable to find image 'p4a:latest' locally
docker: Error response from daemon: pull access denied for p4a, repository does not exist or may require 'docker login'.
See 'docker run --help'.
The command "docker run p4a /bin/sh -c "$COMMAND"" exited with 125.

Log https://travis-ci.org/kivy/python-for-android/jobs/487456672 (will probably disappear when I restart the build).

[AndreMiras]

Status update:
I haven't seen any flaky CI for a while now. Two things I have in mind that could have helped:

• CrystaX support drop, refs Python 3 recipe #1412
• Travis retry use, refs Retry on download hiccups, refs #1306 #1643

[Julian-O]

Given Travis is no longer used and the flaky CI seems to have settled, closing as complete.