Address already in use during tutorial test #11120

marcoabreu · 2018-06-01T07:42:32Z

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-10656/7/pipeline/

+ nosetests-3.4 test_tutorials.py --nologcapture

....[19:25:36] src/io/iter_image_recordio_2.cc:170: ImageRecordIOParser2: ./data/caltech.rec, use 4 threads for decoding..

.......[19:27:51] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

.....[19:28:58] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

..[19:32:06] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

[19:37:22] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

....[19:39:25] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

.[19:39:51] src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...

[19:39:51] src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!

.....[19:40:24] src/operator/././../common/utils.h:416: Optimizer with lazy_update = True detected. Be aware that lazy update with row_sparse gradient is different from standard update, and may lead to different empirical results. See https://mxnet.incubator.apache.org/api/python/optimization/optimization.html for more details.

..[19:40:43] src/operator/././../common/utils.h:416: Optimizer with lazy_update = True detected. Be aware that lazy update with row_sparse gradient is different from standard update, and may lead to different empirical results. See https://mxnet.incubator.apache.org/api/python/optimization/optimization.html for more details.

..Traceback (most recent call last):

  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main

    "__main__", mod_spec)

  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code

    exec(code, run_globals)

  File "/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py", line 16, in <module>

    app.launch_new_instance()

  File "/usr/local/lib/python3.5/dist-packages/traitlets/config/application.py", line 657, in launch_instance

    app.initialize(argv)

  File "<decorator-gen-123>", line 2, in initialize

  File "/usr/local/lib/python3.5/dist-packages/traitlets/config/application.py", line 87, in catch_config_error

    return method(app, *args, **kwargs)

  File "/usr/local/lib/python3.5/dist-packages/ipykernel/kernelapp.py", line 456, in initialize

    self.init_sockets()

  File "/usr/local/lib/python3.5/dist-packages/ipykernel/kernelapp.py", line 248, in init_sockets

    self.control_port = self._bind_socket(self.control_socket, self.control_port)

  File "/usr/local/lib/python3.5/dist-packages/ipykernel/kernelapp.py", line 180, in _bind_socket

    s.bind("tcp://%s:%i" % (self.ip, port))

  File "zmq/backend/cython/socket.pyx", line 547, in zmq.backend.cython.socket.Socket.bind

  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc

zmq.error.ZMQError: Address already in use

F..

======================================================================

FAIL: test_tutorials.test_unsupervised_learning_gan

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/tutorials/test_tutorials.py", line 203, in test_unsupervised_learning_gan

    assert _test_tutorial_nb('unsupervised_learning/gan')

AssertionError: 

-------------------- >> begin captured stdout << ---------------------

Kernel died before replying to kernel_info



--------------------- >> end captured stdout << ----------------------



----------------------------------------------------------------------

Ran 35 tests in 1024.216s



FAILED (failures=1)

build.py: 2018-05-31 19:41:38,780 Running of command in container failed (1): nvidia-docker run --rm -t --shm-size=3g -v /home/jenkins_slave/workspace/it-tutorials-py2:/work/mxnet -v /home/jenkins_slave/workspace/it-tutorials-py2/build:/work/build -u 1001:1001 mxnet/build.ubuntu_gpu /work/runtime_functions.sh tutorialtest_ubuntu_python2_gpu

build.py: 2018-05-31 19:41:38,780 You can try to get into the container by using the following command: nvidia-docker run --rm -t --shm-size=3g -v /home/jenkins_slave/workspace/it-tutorials-py2:/work/mxnet -v /home/jenkins_slave/workspace/it-tutorials-py2/build:/work/build -u 1001:1001 -ti --entrypoint /bin/bash mxnet/build.ubuntu_gpu /work/runtime_functions.sh tutorialtest_ubuntu_python2_gpu

into container: False

Traceback (most recent call last):

  File "ci/build.py", line 307, in <module>

    sys.exit(main())

  File "ci/build.py", line 243, in main

    container_run(platform, docker_binary, shared_memory_size, command)

  File "ci/build.py", line 154, in container_run

    raise subprocess.CalledProcessError(ret, cmd)

subprocess.CalledProcessError: Command 'nvidia-docker run --rm -t --shm-size=3g -v /home/jenkins_slave/workspace/it-tutorials-py2:/work/mxnet -v /home/jenkins_slave/workspace/it-tutorials-py2/build:/work/build -u 1001:1001 mxnet/build.ubuntu_gpu /work/runtime_functions.sh tutorialtest_ubuntu_python2_gpu' returned non-zero exit status 1

script returned exit code 1

Possibly related to hardcoding ports.

The text was updated successfully, but these errors were encountered:

marcoabreu · 2018-06-01T07:42:43Z

@ThomasDelteil

ThomasDelteil · 2018-06-01T15:23:11Z

I tracked this issue for a while, still not sure of why it is happening. My first guess was that the previous notebook run didn't clear the socket in time for the next run.

I added a delay between each notebook run which should have been plenty of time for socket to be released.

I'll do some more digging.

@marcoabreu, how many containers are running per instance concurrently? only one right?

marcoabreu · 2018-06-01T16:30:34Z

They are running concurrently. We got up to 4 containers in parallel. Why should it matter whether a socket is released or not? First of all, networking is virtualized per container, so there should be no problem on that side. Second, we should not require a certain port but just use a random free one.

ThomasDelteil · 2018-06-01T17:08:12Z

The ports are already random, but I don't know whether it checks for availability. I think the issue could be that the ports are not released and rarely but sometimes it happens to try to reuse the same random port.

I have also seen a 'linger=1000', which could mean that the port might be still taken 1s after closing, so ugprading the delay between notebook to 1.1 from 0.5 could also solve the issue.

I'll see if I can make the port deterministically different.

marcoabreu · 2018-06-01T17:43:30Z

I'm not a fan of the delay in between notebooks anyways because it masks a problem. I'd propose that we now remove the delay entirely and track down all the issues coming from that. Otherwise, we're flaky and depend on timing.

ThomasDelteil · 2018-06-01T17:49:25Z

the linger variable is hard coded in the kernelapp.py, if we don't want to use this technique then we need to assign deterministic ports that are different from each other and update the config file of the python notebook between runs.

marcoabreu · 2018-06-01T18:32:23Z

I think that's what we should do. In general, this is required to prepare the path for parallel execution.

access2rohit · 2018-06-26T23:49:52Z

HI Marco,
For reproducing this issue I have to build docs within MXNet since it requires gan.ipynb file. Can you share the setup to reproduce it

marcoabreu · 2018-06-27T22:41:43Z

@ThomasDelteil would you mind assisting here?

anirudh2290 · 2018-06-27T23:44:39Z

assigned to @reminisce. @access2rohit is working on this.

ThomasDelteil · 2018-06-28T09:28:49Z

To reproduce the setup I suggest looking at the jenkins function for tutorial tests. I had put a fix in my last PR before removing them from CI, the issue might be gone already. Can someone start a few hundred runs of the tutorial tests and see if it still happen? Note that they take ~25min so that could take a few days. Actually commenting out most tests except three very fast ones might be a better idea since it isn't related to a specific test, and as one simple test runs in 2-3s with the jupyter kernel overhead. To know which one is fast check the tutorials, some do not much like the NDArray ones. My current best guess is that the issue is related to the fact that the ports used by jupiter internal mechanism are chosen randomly and that there is linger=1000 hard-coded in the jupiter code somewhere that keep it being used for 1sec. For every test there is ~1/10000 chance that the same port will be reused (3 ports are picked between 1-100000), which makes it 1/300 because we have 30 tests and 1/150 because we run on python2 and python3. That seems roughly consistent with the number of reports we've had, about once every 150 CI runs. There is no easy way to set the ports to fixed deterministic value. My latest fix added a non-ideal 1.1 sleep between tests. Let's see if that fixed it. The above explanation might be bogus too. I'm on my phone in a plane and can assist more from Friday onwards. Thanks for looking into it @reminisce <https://github.com/reminisce> and @access2rohit <https://github.com/access2rohit>!

…

On Wed, Jun 27, 2018, 20:45 Anirudh Subramanian ***@***.***> wrote: assigned to @reminisce <https://github.com/reminisce> @access2rohit <https://github.com/access2rohit> is working on this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11120 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADi001F-ydKxRBX2r1PYR-DJXVfJv14lks5uBBkDgaJpZM4UWPoF> .

marcoabreu · 2018-06-28T09:32:44Z

Just to be on the safe side: please try what Thomas suggested only locally and don't submit these jobs to CI. This test suite causes resource exhaustion on our CI and running so many in parallel would basically knock off our instances. Thomas Delteil <[email protected]> schrieb am Do., 28. Juni 2018, 11:29:

…

To reproduce the setup I suggest looking at the jenkins function for tutorial tests. I had put a fix in my last PR before removing them from CI, the issue might be gone already. Can someone start a few hundred runs of the tutorial tests and see if it still happen? Note that they take ~25min so that could take a few days. Actually commenting out most tests except three very fast ones might be a better idea since it isn't related to a specific test, and as one simple test runs in 2-3s with the jupyter kernel overhead. To know which one is fast check the tutorials, some do not much like the NDArray ones. My current best guess is that the issue is related to the fact that the ports used by jupiter internal mechanism are chosen randomly and that there is linger=1000 hard-coded in the jupiter code somewhere that keep it being used for 1sec. For every test there is ~1/10000 chance that the same port will be reused (3 ports are picked between 1-100000), which makes it 1/300 because we have 30 tests and 1/150 because we run on python2 and python3. That seems roughly consistent with the number of reports we've had, about once every 150 CI runs. There is no easy way to set the ports to fixed deterministic value. My latest fix added a non-ideal 1.1 sleep between tests. Let's see if that fixed it. The above explanation might be bogus too. I'm on my phone in a plane and can assist more from Friday onwards. Thanks for looking into it @reminisce <https://github.com/reminisce> and @access2rohit <https://github.com/access2rohit>! On Wed, Jun 27, 2018, 20:45 Anirudh Subramanian ***@***.***> wrote: > assigned to @reminisce <https://github.com/reminisce> @access2rohit > <https://github.com/access2rohit> is working on this. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #11120 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/ADi001F-ydKxRBX2r1PYR-DJXVfJv14lks5uBBkDgaJpZM4UWPoF > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11120 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARxB60DSOdndKcaobWgfxBGzIAMRSv4Dks5uBKH-gaJpZM4UWPoF> .

access2rohit · 2018-07-11T23:24:51Z

@ThomasDelteil : Can you specify which "jenkins functions for tutorial tests" are you referring to in order to reproduce the tests ?

marcoabreu · 2018-07-12T09:59:48Z

They are defined here: https://github.com/apache/incubator-mxnet/blob/master/ci/docker/runtime_functions.sh#L580

Disable was in #11170

ThomasDelteil · 2018-07-17T21:30:43Z

@marcoabreu the suite was run 300 times and the address already in use wasn't triggered. Can we close this issue?

access2rohit · 2018-07-20T17:56:02Z

@marcoabreu The address in use issue not being triggered by running the suite couple hundred times. Should this issue be closed ? If not then can you shared the exact setup to reproduce it. I worked with @ThomasDelteil to get these tests to run and since they take time to complete a single run I ran it 300 times in order to reproduce it but it still succeeded.

marcoabreu · 2018-07-25T10:18:37Z

Did you remove the thread.sleep you introduced to work around this error?

The environment is the regular CI pipeline.

I'll leave it up to you, Thomas.

ThomasDelteil · 2018-07-25T16:58:28Z

The sleep is still there and I think it is necessary. It doesn't bring much extra delay. 30s on a 30min test run, that 1.6% extra delay, which is acceptable.

I would close it, thanks.

marcoabreu · 2018-07-25T17:25:36Z

The question is whether we are masking a problem with that sleep. It feels like fixing a race condition by adding a sleep.

ThomasDelteil · 2018-07-25T17:30:47Z

there is a hardcoded 1000ms linger on the socket in the base code of Jupyter, which I believe is the root cause of the problem.

We can fork jupyter or monkey patch it but I think the better trade-off with performance and maintainability is the sleep.

marcoabreu · 2018-07-25T17:36:38Z

Okay, that makes sense. Please note that even with the increased delay we sometimes experienced this error.

For reproducing, please run multiple instances of the tests in parallel to see if the error occurs.

We can easily increase the delay to more than a second if that fixes the problem.

ThomasDelteil · 2018-07-25T19:19:09Z

To clarify: I had first put a 0.5s delay as a wild guess when we still had the error, I only then found out about the 1000ms socket linger, increased delay to 1.1s but the test was disabled already at that point. access2rohit tested that version and didn't reproduce the bug.

marcoabreu · 2018-07-25T19:29:17Z

Oh, I was under the impression that we disabled the test after adding the 1.1s delay because it was still failing and we didn't know why because it should have worked due to the 1s timeout.

ThomasDelteil · 2018-07-25T21:05:58Z

See https://github.com/apache/incubator-mxnet/pull/11170/files

marcoabreu · 2018-07-25T21:09:57Z

Great, thanks a lot for the link Thomas!

Can we bring the tutorial tests into nightly and then close this issue? Currently, this issue serves as master ticket to mark the tests as disabled

ThomasDelteil · 2018-07-25T23:03:55Z

Sure, it's in the works by @vishaalkapoor

marcoabreu · 2018-08-03T10:43:50Z

Happened again: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTests_onBinaries/detail/NightlyTests_onBinaries/102/pipeline

+ nosetests-3.4 --with-xunit --xunit-file nosetests_straight_dope_python3_single_gpu.xml test_notebooks_single_gpu.py --nologcapture

.......[01:16:58] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:109: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

.....[01:28:00] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:109: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

[01:28:00] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:109: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

.Traceback (most recent call last):

  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main

    "__main__", mod_spec)

  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code

    exec(code, run_globals)

  File "/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py", line 16, in <module>

    app.launch_new_instance()

  File "/usr/local/lib/python3.5/dist-packages/traitlets/config/application.py", line 657, in launch_instance

    app.initialize(argv)

  File "<decorator-gen-123>", line 2, in initialize

  File "/usr/local/lib/python3.5/dist-packages/traitlets/config/application.py", line 87, in catch_config_error

    return method(app, *args, **kwargs)

  File "/usr/local/lib/python3.5/dist-packages/ipykernel/kernelapp.py", line 456, in initialize

    self.init_sockets()

  File "/usr/local/lib/python3.5/dist-packages/ipykernel/kernelapp.py", line 238, in init_sockets

    self.shell_port = self._bind_socket(self.shell_socket, self.shell_port)

  File "/usr/local/lib/python3.5/dist-packages/ipykernel/kernelapp.py", line 180, in _bind_socket

    s.bind("tcp://%s:%i" % (self.ip, port))

  File "zmq/backend/cython/socket.pyx", line 549, in zmq.backend.cython.socket.Socket.bind

  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc

zmq.error.ZMQError: Address already in use

ERROR:root:Kernel died before replying to kernel_info

F...........................[01:48:50] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:109: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

.

======================================================================

FAIL: test_generative_adversarial_networks (test_notebooks_single_gpu.StraightDopeSingleGpuTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/work/mxnet/tests/nightly/straight_dope/test_notebooks_single_gpu.py", line 274, in test_generative_adversarial_networks

    assert _test_notebook('chapter14_generative-adversarial-networks/conditional')

AssertionError



----------------------------------------------------------------------

Ran 42 tests in 2291.515s



FAILED (failures=1)

build.py: 2018-08-03 01:49:31,003 Running of command in container failed (1):

nvidia-docker\

	run\

	--rm\

	-t\

	--shm-size=500m\

	-v\

	/home/jenkins_slave/workspace/straight_dope-single_gpu:/work/mxnet\

	-v\

	/home/jenkins_slave/workspace/straight_dope-single_gpu/build:/work/build\

	-v\

	/tmp/ci_ccache:/work/ccache\

	-u\

	1001:1001\

	-e\

	CCACHE_MAXSIZE=500G\

	-e\

	CCACHE_TEMPDIR=/tmp/ccache\

	-e\

	CCACHE_DIR=/work/ccache\

	-e\

	CCACHE_LOGFILE=/tmp/ccache.log\

	mxnetci/build.ubuntu_nightly_gpu\

	/work/runtime_functions.sh\

	nightly_straight_dope_python3_single_gpu_tests



build.py: 2018-08-03 01:49:31,003 You can get into the container by adding the -i option

Traceback (most recent call last):

  File "ci/build.py", line 408, in <module>

    sys.exit(main())

  File "ci/build.py", line 337, in main

    local_ccache_dir=args.ccache_dir, interactive=args.interactive)

  File "ci/build.py", line 224, in container_run

    raise subprocess.CalledProcessError(ret, cmd)

subprocess.CalledProcessError: Command 'nvidia-docker\

	run\

	--rm\

	-t\

	--shm-size=500m\

	-v\

	/home/jenkins_slave/workspace/straight_dope-single_gpu:/work/mxnet\

	-v\

	/home/jenkins_slave/workspace/straight_dope-single_gpu/build:/work/build\

	-v\

	/tmp/ci_ccache:/work/ccache\

	-u\

	1001:1001\

	-e\

	CCACHE_MAXSIZE=500G\

	-e\

	CCACHE_TEMPDIR=/tmp/ccache\

	-e\

	CCACHE_DIR=/work/ccache\

	-e\

	CCACHE_LOGFILE=/tmp/ccache.log\

	mxnetci/build.ubuntu_nightly_gpu\

	/work/runtime_functions.sh\

	nightly_straight_dope_python3_single_gpu_tests' returned non-zero exit status 1

script returned exit code 1

ThomasDelteil · 2018-08-03T15:10:24Z

Looks like the sleep(1.1) got removed as far as I can see. edit: it's not, investigating with Vishaal. It seems like we might have misdiagnosed this error, it happens consistently when a notebook crashes.

…

On Fri, Aug 3, 2018, 03:44 Marco de Abreu ***@***.***> wrote: Happened again: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTests_onBinaries/detail/NightlyTests_onBinaries/102/pipeline — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11120 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADi000XEYmMEPAqZvOtRcgpBiLfzZgI_ks5uNCmNgaJpZM4UWPoF> .

The tutorial tests were disabled due to a flaky port acquisition strategy (apache#11170). The issue has been remedied (apache#11120). This change re-enables the tutorial tests in the Nightly test suite.

vishaalkapoor · 2018-11-06T04:08:54Z

Notes:

The address already in use issue is resolved by this [MXAPPS-805] Notebook execution failures in CI. #12068 merged Aug 10.
The tutorial tests will be re-enabled in [MXNET-1194] Reenable nightly tutorials tests for Python2 and Python3 #13099 in review.
Once [MXNET-1194] Reenable nightly tutorials tests for Python2 and Python3 #13099 is merged this issue can be closed.

srochel · 2018-11-10T04:34:43Z

#13099 is merged, please close.
Why did we not use mechanism describe in https://blog.github.com/2013-05-14-closing-issues-via-pull-requests/ i.e. add fixes #13099 in PR? Are people not aware of it?

marcoabreu added Test Flaky labels Jun 1, 2018

anirudh2290 assigned reminisce Jun 27, 2018

anirudh2290 added the Disabled test label Jul 12, 2018

marcoabreu closed this as completed Nov 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Address already in use during tutorial test #11120

Address already in use during tutorial test #11120

marcoabreu commented Jun 1, 2018

marcoabreu commented Jun 1, 2018

ThomasDelteil commented Jun 1, 2018

marcoabreu commented Jun 1, 2018 •

edited

Loading

ThomasDelteil commented Jun 1, 2018

marcoabreu commented Jun 1, 2018

ThomasDelteil commented Jun 1, 2018 •

edited

Loading

marcoabreu commented Jun 1, 2018

access2rohit commented Jun 26, 2018

marcoabreu commented Jun 27, 2018

anirudh2290 commented Jun 27, 2018 •

edited

Loading

ThomasDelteil commented Jun 28, 2018 via email

marcoabreu commented Jun 28, 2018 via email

access2rohit commented Jul 11, 2018

marcoabreu commented Jul 12, 2018 •

edited

Loading

ThomasDelteil commented Jul 17, 2018 •

edited

Loading

access2rohit commented Jul 20, 2018

marcoabreu commented Jul 25, 2018

ThomasDelteil commented Jul 25, 2018

marcoabreu commented Jul 25, 2018

ThomasDelteil commented Jul 25, 2018

marcoabreu commented Jul 25, 2018

ThomasDelteil commented Jul 25, 2018 •

edited

Loading

marcoabreu commented Jul 25, 2018

ThomasDelteil commented Jul 25, 2018

marcoabreu commented Jul 25, 2018 •

edited

Loading

ThomasDelteil commented Jul 25, 2018

marcoabreu commented Aug 3, 2018 •

edited

Loading

ThomasDelteil commented Aug 3, 2018 via email •

edited

Loading

vishaalkapoor commented Nov 6, 2018

srochel commented Nov 10, 2018

Address already in use during tutorial test #11120

Address already in use during tutorial test #11120

Comments

marcoabreu commented Jun 1, 2018

marcoabreu commented Jun 1, 2018

ThomasDelteil commented Jun 1, 2018

marcoabreu commented Jun 1, 2018 • edited Loading

ThomasDelteil commented Jun 1, 2018

marcoabreu commented Jun 1, 2018

ThomasDelteil commented Jun 1, 2018 • edited Loading

marcoabreu commented Jun 1, 2018

access2rohit commented Jun 26, 2018

marcoabreu commented Jun 27, 2018

anirudh2290 commented Jun 27, 2018 • edited Loading

ThomasDelteil commented Jun 28, 2018 via email

marcoabreu commented Jun 28, 2018 via email

access2rohit commented Jul 11, 2018

marcoabreu commented Jul 12, 2018 • edited Loading

ThomasDelteil commented Jul 17, 2018 • edited Loading

access2rohit commented Jul 20, 2018

marcoabreu commented Jul 25, 2018

ThomasDelteil commented Jul 25, 2018

marcoabreu commented Jul 25, 2018

ThomasDelteil commented Jul 25, 2018

marcoabreu commented Jul 25, 2018

ThomasDelteil commented Jul 25, 2018 • edited Loading

marcoabreu commented Jul 25, 2018

ThomasDelteil commented Jul 25, 2018

marcoabreu commented Jul 25, 2018 • edited Loading

ThomasDelteil commented Jul 25, 2018

marcoabreu commented Aug 3, 2018 • edited Loading

ThomasDelteil commented Aug 3, 2018 via email • edited Loading

vishaalkapoor commented Nov 6, 2018

srochel commented Nov 10, 2018

marcoabreu commented Jun 1, 2018 •

edited

Loading

ThomasDelteil commented Jun 1, 2018 •

edited

Loading

anirudh2290 commented Jun 27, 2018 •

edited

Loading

marcoabreu commented Jul 12, 2018 •

edited

Loading

ThomasDelteil commented Jul 17, 2018 •

edited

Loading

ThomasDelteil commented Jul 25, 2018 •

edited

Loading

marcoabreu commented Jul 25, 2018 •

edited

Loading

marcoabreu commented Aug 3, 2018 •

edited

Loading

ThomasDelteil commented Aug 3, 2018 via email •

edited

Loading