Retry uploads on (connection) errors #158

blueyed · 2018-09-07T16:44:19Z

Example: https://ci.appveyor.com/project/pytestbot/pytest/build/1.0.5141/job/ycpyxb9yvrckho21#L402 (RemoteDisconnected)

codecov-bash tries 3 times already, sleeping for 30s inbetween.

blueyed · 2018-11-17T14:40:00Z

Another example:

      _____          _
     / ____|        | |
    | |     ___   __| | ___  ___ _____   __
    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
    | |___| (_) | (_| |  __/ (_| (_) \ V /
     \_____\___/ \____|\___|\___\___/ \_/
                                    v2.0.15
==> Detecting CI provider
    AppVeyor Detected
    Fixing merge commit SHA
==> Preparing upload
    Error running `cd 'C:\projects\pytest' && git ls-files`: None
XX> Skip processing gcov
==> Collecting reports
XX> Searching for reports disabled
    Targeting specific files
    + coverage.xml bytes=1071184
==> Appending environment variables
    + TOXENV
==> Uploading
    .url https://codecov.io
    .query commit=3acfa3abdc7577f97bb40b9790c10d43c0e486e4&branch=master&job=pytestbot%2Fpytest%2F1.0.5975&pr=4409&service=appveyor&build=vku9wlsxtbjwwk0j&slug=pytest-dev%2Fpytest&flags=py27%2Cwindows&package=py2.0.15
    Pinging Codecov...
Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
Tip: See all example repositories: https://github.com/codecov?query=example
Support channels:
  Email:   [email protected]
  IRC:     #codecov
  Gitter:  https://gitter.im/codecov/support
  Twitter: @codecov

blueyed · 2019-02-18T16:19:26Z

Any update on this?

Another example (from AppVeyor):

==> Uploading
    .url https://codecov.io
    .query commit=7e9f35e6cba84c386d29fc45e28d60b64927f753&branch=master&job=pytestbot%2Fpytest%2F1.0.6650&pr=4799&service=appveyor&build=r1cx4er7tmmd98ul&slug=pytest-dev%2Fpytest&package=py2.0.15
    Pinging Codecov...
Error: HTTPSConnectionPool(host='codecov.io', port=443): Max retries exceeded with url: /upload/v4?commit=7e9f35e6cba84c386d29fc45e28d60b64927f753&branch=master&job=pytestbot%2Fpytest%2F1.0.6650&pr=4799&service=appveyor&build=r1cx4er7tmmd98ul&slug=pytest-dev%2Fpytest&package=py2.0.15 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x02DFC030>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))

Tip: See all example repositories: https://github.com/codecov?query=example
Support channels:
  Email:   [email protected]
  IRC:     #codecov
  Gitter:  https://gitter.im/codecov/support
  Twitter: @codecov

Sorry, we tried running command for 3 times and all attempts were unsuccessful!

quantus · 2019-07-23T16:21:20Z

I've also seen quite a few similar connection errors from our CircleCI builds. My current solution for the issue has been running codecov like this:

codecov --required || (sleep 5 && codecov --required) || (sleep 5 && codecov --required) || (sleep 5 && codecov --required) || (sleep 5 && codecov --required)

I know it isn't pretty but I haven't seen it failing yet.

timandy · 2019-08-20T08:48:32Z

I got the same issue
Could add a parameter to specify the number of retries?

1748      _____          _
1749     / ____|        | |
1750    | |     ___   __| | ___  ___ _____   __
1751    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
1752    | |___| (_) | (_| |  __/ (_| (_) \ V /
1753     \_____\___/ \____|\___|\___\___/ \_/
1754                                    v2.0.15
1755
1756==> Detecting CI provider
1757    Travis Detected
1758==> Preparing upload
1759==> Processing gcov (disable by -X gcov)
1760    Executing gcov (find /home/travis/build/timandy/linq -not -path './bower_components/**' -not -path './node_modules/**' -not -path './vendor/**' -type f -name '*.gcno'  -exec gcov -pb  {} +)
1761==> Collecting reports
1762    + /home/travis/build/timandy/linq/target/site/jacoco/jacoco.xml bytes=1354762
1763==> Appending environment variables
1764    + TRAVIS_JDK_VERSION
1765    + TRAVIS_OS_NAME
1766==> Uploading
1767    .url https://codecov.io
1768    .query pr=false&service=travis&package=py2.0.15&job=573661452&build=230.1&branch=master&commit=f768b4f9b4b1cd2acdf5152d0f7be269a22793c0&slug=timandy%2Flinq
1769    Pinging Codecov...
1770    Uploading to S3...
1771Error: HTTPSConnectionPool(host='codecov.io', port=443): Max retries exceeded with url: /codecov/v4/raw/2019-08-19/BD907C23C2C05F757F59BF33F38DF67D/f768b4f9b4b1cd2acdf5152d0f7be269a22793c0/7d85c4bc-4456-43cd-9289-81a22b1c23b1.txt?AWSAccessKeyId=AKIAIHLZSCQCS4WIHD4A&Expires=1566195120&Signature=Hj%2BcQ2c4lj7zLAMWUejjidNN6DU%3D (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb360244390>: Failed to establish a new connection: [Errno 110] Connection timed out',))
1772
1773Tip: See an example jdk repo: https://github.com/codecov/example-jdk
1774Support channels:
1775  Email:   [email protected]
1776  IRC:     #codecov
1777  Gitter:  https://gitter.im/codecov/support
1778  Twitter: @codecov
1779
1780
1781

…odecov/codecov-python#158

ikostan · 2020-04-20T09:38:08Z

Hi all.
Same issue, using GitLab CI:

Uploading to S3... Error: HTTPSConnectionPool(host='codecov.io', port=443): Max retries exceeded with url: /codecov/v4/raw/2020-04-20/102BD5852C808ECE77423D85935DC840/fc50b3d848de669df5679d7f36fbd3b14e460e2d/88c1c006-6219-41d0-8a6e-8dc3709b6235.txt?AWSAccessKeyId=AKIAIHLZSCQCS4WIHD4A&Expires=1587375167&Signature=ycNO0ZmrV%2FVhr2jmGd3dIi7Juc4%3D (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0b00bfb6d8>: Failed to establish a new connection: [Errno 110] Connection timed out',)) Tip: See all example repositories: https://github.com/codecov?query=example Support channels: Email: [email protected] IRC: #codecov Gitter: https://gitter.im/codecov/support Twitter: @codecov

jquast · 2020-05-29T15:26:05Z

Dear maintainer(s): Would a PR to use a 3rd party "retryable" decorator, https://tenacity.readthedocs.io/en/latest/ be accepted?

nmoinvaz · 2020-05-29T15:29:05Z

I implemented a retry for requests here and retry number is configurable via args:

https://github.com/nmoinvaz/codecov-python/commit/9632dc655ed378ece67bf54861a300dd1e732aef
https://github.com/nmoinvaz/codecov-python/commit/30ff5e3140291b1a11bea9f2973efae72d8dd2aa

If this is something that would be accepted I can submit a PR.

xoxys · 2020-06-03T12:35:37Z

Looks like this is a long-running issue. What about the solution from @nmoinvaz? Could it be added or not? What is needed to finally get a solution into codecov-python?

blueyed · 2020-08-31T18:47:18Z

Retries have been added in 07583f3 by @thomasrockhu, thanks!

However I've still seen this now:

      _____          _
     / ____|        | |
    | |     ___   __| | ___  ___ _____   __
    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
    | |___| (_) | (_| |  __/ (_| (_) \ V /
     \_____\___/ \____|\___|\___\___/ \_/
                                    v2.1.9
==> Detecting CI provider
    Gitlab CI Detected
==> Preparing upload
XX> Skip processing gcov
==> Collecting reports
    Targeting specific files
    + coverage.xml bytes=1272194
==> Appending environment variables
    + TOXENV
==> Uploading
    .url https://codecov.io
    .query commit=XXX
    Gzipping contents..
    Compressed contents to 86377 bytes
    Pinging Codecov...
Error: HTTPSConnectionPool(host='codecov.io', port=443): Max retries exceeded with url: /upload/v4?commit=XXX (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000026EC415FE48>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))

From this it is not clear if there have been any retries at all (should get logged probably), and it might have worked better with a delay between retries (#293, #295).

nmoinvaz · 2020-08-31T20:22:17Z

codecov-bash tries 3 times already, sleeping for 30s inbetween.

Looking at codecov bash uploader it appears to now try 5 times, sleeping for 2s in between.

nmoinvaz · 2020-12-02T06:15:35Z

I believe this issue is resolved now. It does delay between retries and also displays each time it retries.

In version 1.18.0 and earlier, >> term.split_seqs(term.move_right(333) + 'xyz', maxsplit=1) ['\x1b[333C', '333', 'xyz'] This is a bug, it duplicates the matched parameter, this is now corrected: ['\x1b[3C', 'xyz'] Previously, we documented "same arguments as "re.split", so we must also implement maxsplit and flags. Also, - fix flake8 linting of tests by moving fixtures to conftest.py, fixes "unused" or "re-definition from import" errors. - version stamp blessed/__init__.py like a codegen step i guess - remove run_codecov.py, its been fixed upstream codecov/codecov-python#158 (comment)

* Bugfix for term.split_seqs('term.right(3)') In version 1.18.0 and earlier, >> term.split_seqs(term.move_right(333) + 'xyz', maxsplit=1) ['\x1b[333C', '333', 'xyz'] This is a bug, it duplicates the matched parameter, this is now corrected: ['\x1b[3C', 'xyz'] Previously, we documented "same arguments as "re.split", so we must also implement maxsplit and flags. Also, - fix flake8 linting of tests by moving fixtures to conftest.py, fixes "unused" or "re-definition from import" errors. - version stamp blessed/__init__.py like a codegen step i guess - remove run_codecov.py, its been fixed upstream codecov/codecov-python#158 (comment)

blueyed mentioned this issue Sep 7, 2018

Amend CHANGELOG with missing #3251 pytest-dev/pytest#3948

Merged

This was referenced Nov 17, 2018

retry #171

Closed

Improve CI / testing Python-Markdown/markdown#754

Merged

kapilt mentioned this issue Jun 25, 2019

Incomplete read error #196

Closed

friedbunny mentioned this issue Jul 29, 2019

codecov.io is flaky and can fail builds mapbox/mapbox-gl-native#15095

Closed

pquentin mentioned this issue Aug 21, 2019

Bump pytest from 5.0.1 to 5.1.1 python-trio/trio#1196

Merged

Niccolum added a commit to Niccolum/intrst_algrms that referenced this issue Sep 1, 2019

add few changes to travisCI for not to error to request. Here more: c…

b4642cd

…odecov/codecov-python#158

yanokwa mentioned this issue Sep 10, 2019

Run codecov in subshell to prevent blocking build XLSForm/pyxform#363

Merged

lilithxxx mentioned this issue Sep 21, 2019

Added new structures dummy data generator oppia/oppia#7621

Merged

9 tasks

seanlip mentioned this issue Sep 21, 2019

Try to ensure that Codecov always uploads a coverage report. oppia/oppia#7656

Closed

raczben mentioned this issue Feb 4, 2020

Unable to upload from AppVeyor #170

Closed

jquast mentioned this issue May 29, 2020

use tenacity, move run_codecov.py out of view jquast/blessed#171

Closed

sbdchd mentioned this issue Jun 19, 2020

add(tools): code coverage reporting for bot & web_api chdsbd/kodiak#443

Merged

kjulian3 mentioned this issue Jul 2, 2020

codecov.io is flaky NeuralNetworkVerification/Marabou#318

Closed

rly mentioned this issue Jul 9, 2020

Work around codecov.io connection failures hdmf-dev/hdmf#396

Closed

5 tasks

v1v mentioned this issue Jul 21, 2020

[CI]: retry with sleep elastic/beats#20065

Merged

blueyed mentioned this issue Aug 31, 2020

Wait between retries #293

Closed

thomasrockhu self-assigned this Nov 29, 2020

jquast mentioned this issue Mar 6, 2021

Bugfix for term.split_seqs('term.right(3)') #197 jquast/blessed#199

Merged

thomasrockhu removed their assignment Jul 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retry uploads on (connection) errors #158

Retry uploads on (connection) errors #158

blueyed commented Sep 7, 2018

blueyed commented Nov 17, 2018

blueyed commented Feb 18, 2019

quantus commented Jul 23, 2019

timandy commented Aug 20, 2019 •

edited

Loading

ikostan commented Apr 20, 2020 •

edited

Loading

jquast commented May 29, 2020

nmoinvaz commented May 29, 2020 •

edited

Loading

xoxys commented Jun 3, 2020

blueyed commented Aug 31, 2020

nmoinvaz commented Aug 31, 2020

nmoinvaz commented Dec 2, 2020

Retry uploads on (connection) errors #158

Retry uploads on (connection) errors #158

Comments

blueyed commented Sep 7, 2018

blueyed commented Nov 17, 2018

blueyed commented Feb 18, 2019

quantus commented Jul 23, 2019

timandy commented Aug 20, 2019 • edited Loading

ikostan commented Apr 20, 2020 • edited Loading

jquast commented May 29, 2020

nmoinvaz commented May 29, 2020 • edited Loading

xoxys commented Jun 3, 2020

blueyed commented Aug 31, 2020

nmoinvaz commented Aug 31, 2020

nmoinvaz commented Dec 2, 2020

timandy commented Aug 20, 2019 •

edited

Loading

ikostan commented Apr 20, 2020 •

edited

Loading

nmoinvaz commented May 29, 2020 •

edited

Loading