setup_rosdep is unstable

[mathias-luedtke]

Currently, rosdep update seems to fail a lot with messages like this:

ERROR: unable to process source [https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/base.yaml]:
	<urlopen error ('_ssl.c:710: The handshake operation timed out',)> (https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/base.yaml)

At the moment, I would suggest to monitor this issue and restart jobs a lot :-/
We can't do much except for increasing the number of retries (currently just 2 tries in total).

[ross-desmond]

I've added a ticket here as well for rosdep ci builds: ros-infrastructure/rosdep#677

[gavanderhoorn]

A bit of a "me too", but I just ran into this as well.

Do we have any mitigation in place?

[mathias-luedtke]

Do we have any mitigation in place?

We could increase the number of retries.
But I have restarted one of my builds >10 times now and still no luck.
It looks like catkin_lint has trouble downloading as well.

it works on my laptop, so I guess it is a Travis connectivity issue

[gavanderhoorn]

Do we have any mitigation in place?

We could increase the number of retries.
But I have restarted one of my builds >10 times now and still no luck.

Yes, I've restarted a build a large nr of times as well and no change.

it works on my laptop, so I guess it it as Travis connectivity issue

hm. That is interesting info actually.

[ipa-rmb]

Same problem here, I sometimes need to restart so many times until something works.

[130s]

This has been happening on Gitlab CI too :/

it works on my laptop, so I guess it it as Travis connectivity issue

hm. That is interesting info actually.

Running Python version is different? Just a guess.

[ross-desmond]

We (seems like a community effort) have posted support issues with travis and they have come up with a workaround as they investigate the failures.

https://www.traviscistatus.com/incidents/kyf149kl6bvp?u=3g7kt113nmgs

See discussion here:
https://travis-ci.community/t/ssl-the-handshake-operation-timed-out-to-github/3288/2

[mathias-luedtke]

@ross-desmond: Thanks for pushing this!

According to the incident description it should help to switch to switch to host network mode, e.g.
DOCKER_BUILD_OPTS='--network=host' and DOCKER_RUN_OPTS='--network=host'

It seems to work.
If the problem persists, we could inject these settings automatically..
(only for the automated CI scripts!)

[mathias-luedtke]

This has been happening on Gitlab CI too :/

yep, https://gitlab.com/ipa-mdl/industrial_ci/pipelines
seems to work again, at least on some runners (e.g shared-runners-manager-4.gitlab.com and shared-runners-manager-6.gitlab.com).

[mathias-luedtke]

I have now deployed the hotfix for Travis.
Prerelease tests might still be affected.

[fmessmer]

thanks for the efforts done so far...seems like the handshake issue is indeed resolved with the hotfix...

---

however, I'm still facing quite some pip install failures

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-FnrUmL/aiohttp/
ERROR: the following rosdeps failed to install
  pip: command [pip install -U -q slackclient] failed

(seems similar to what is reported here: https://travis-ci.community/t/ssl-the-handshake-operation-timed-out-to-github/3288/2)

Any idea whether this is something different? Or how it can be resolved?

Never mind! It is a particular issue with slackclient that I could also reproduce running industrial_ci locally on my laptop...still, I have no clue how to solve it 😉

[mathias-luedtke]

slackclient requires Python3.6, rosdep is using pip(=pip2 on Ubuntu).
pip installs v1.3.1 in this case.

[mathias-luedtke]

I have now deployed the hotfix for Travis.

Apparently the hotfix is not needed anymore (#366).
Not sure if it was fixed by Travis CI or Github (https://www.githubstatus.com/incidents/chq6jdw9x8t5).
@130s: Do you still have this problem with your Gitlab builds?

[130s]

I don't know what was the fix, but on some of our Gitlab repo where we had this issue we added the same things as #365 and the issue stopped (haven't tried without that change since then).

[mathias-luedtke]

haven't tried without that change since then

It would be great if you (or somebody else) could test this again.

[mathias-luedtke]

Seems to be resolved