Allow user to replace ingress network #31714

aboch · 2017-03-09T22:58:11Z

Vendoring of swarmkit carries a change to support ingress network replacement and two important scheduler fixes:

Allow replacing ingress network Allow replacing ingress network swarmkit#2028
scheduler: Fix accounting when task ends up in multiple groups scheduler: Fix accounting when task ends up in multiple groups swarmkit#2031
scheduler: Correctly count tasks at bottom level of decision tree scheduler: Correctly count tasks at bottom level of decision tree swarmkit#2032

Vendoring of libnetwork carries:

change related to ingress network programmability Expose ingress network option libnetwork#1678
change to allow user to control container's interfaces prefix on bridge networks Support for com.docker.network.container_iface_prefix driver label libnetwork#1667

This PR allows user to remove and (re)create the ingress network:

A fresh new cluster will come up with the default ingress network as today.
The ingress network can be removed and never be recreated.

$ docker network rm ingress
WARNING! Before removing the routing-mesh network, make sure all the nodes in your swarm run the same docker engine version. Otherwise, removal may not be effective and functionality of newly create ingress networks will be impaired.
Are you sure you want to continue? [y/N]
ingress

On creation, user can specify any parameter as for any other swarm network. Name can be anything.

$ docker network create -d overlay --opt com.docker.network.mtu=1200 --ingress my-ingress
syws4fv0sqrwkhblly297z569

The ingress network removal requires that no services depending on the network exist.

$ docker network rm my-ingress 
WARNING! Before removing the routing-mesh network, make sure all the nodes in your swarm run the same docker engine version. Otherwise, removal may not be effective and functionality of newly create ingress networks will be impaired.
Are you sure you want to continue? [y/N]
Error response from daemon: rpc error: code = 7 desc = ingress network cannot be removed because service lbxksk0633cqzbqrlnwx9v4kp depends on it

Swarm services creation/update will make sure the ingress network is present before succeeding, if the service requires the routing mesh.

$ docker service create --name srv-on-rmesh -p 9000:8000 busybox top
Error response from daemon: rpc error: code = 7 desc = service needs ingress network, but ingress network is not present
$
$ docker service update --publish-add 7000:80 s0
Error response from daemon: rpc error: code = 7 desc = service needs ingress network, but ingress network is not present

Fixes #24220

Depends on moby/swarmkit/pull/2028 and moby/libnetwork/pull/1678

- A picture of a cute animal (not mandatory but encouraged)

vdemeester · 2017-03-13T15:25:54Z

/cc @thaJeztah @cpuguy83 @aaronlehmann @mavenugo

thaJeztah · 2017-03-13T15:32:20Z

Is the custom ingress network still treated "special", i.e. is "icc" communication disabled by default?

aboch · 2017-03-13T15:43:44Z

@thaJeztah

is "icc" communication disabled by default

That has not changed, even now icc is not disabled on the ingress network.

What container cannot do today (and this PR does not change that) is to reach services via the ingress network on L4 ports other than the ones the services exposes. That is achieved internally, without user configurations needed on the ingress network creation.

mavenugo · 2017-03-13T15:48:42Z

@aboch but hopefully Service Discovery, VIP and LB are disabled on this Ingress network ?

aboch · 2017-03-13T15:50:12Z

but Service Discovery, VIP and LB are disabled on this Ingress network correct ?

Yes, forgot that thanks. That is all disabled. It's all done internally and user does not and will not need to control it via network creation options.

aboch · 2017-03-13T21:34:36Z

hmm some mess happening with vendoring. let me look into that.

Looks like I need to update my vndr tool.

Edit: run the updted vndr, should be fine now

aboch · 2017-03-14T13:57:40Z

hmm, a docker-py integration test time out failed janky (https://jenkins.dockerproject.org/job/Docker-PRs/40326/console):

07:08:00 ../../../../../docker-py/tests/integration/api_container_test.py ...s........sx.................Build timed out (after 120 minutes). Marking the build as failed.
07:08:00 Build timed out (after 120 minutes). Marking the build as aborted.

But when I run it locally, the docker-py integ test passes:

============================= test session starts ==============================
platform linux2 -- Python 2.7.9, pytest-2.9.1, py-1.4.32, pluggy-0.3.1
rootdir: /docker-py, inifile: pytest.ini
plugins: cov-2.1.0
collected 239 items

../../../../../docker-py/tests/integration/api_build_test.py ........s.
../../../../../docker-py/tests/integration/api_client_test.py ........
../../../../../docker-py/tests/integration/api_container_test.py ...s........sx...........................................s.....
../../../../../docker-py/tests/integration/api_exec_test.py ........
../../../../../docker-py/tests/integration/api_healthcheck_test.py ...
../../../../../docker-py/tests/integration/api_image_test.py ..........s..s
../../../../../docker-py/tests/integration/api_network_test.py .........s............s.
../../../../../docker-py/tests/integration/api_plugin_test.py ssssssssss
../../../../../docker-py/tests/integration/api_secret_test.py sssss
../../../../../docker-py/tests/integration/api_service_test.py ..........ss.s......s
../../../../../docker-py/tests/integration/api_swarm_test.py ..s........
../../../../../docker-py/tests/integration/api_volume_test.py ..s...s..
../../../../../docker-py/tests/integration/client_test.py ...
../../../../../docker-py/tests/integration/errors_test.py .
../../../../../docker-py/tests/integration/models_containers_test.py .......................
../../../../../docker-py/tests/integration/models_images_test.py .x....
../../../../../docker-py/tests/integration/models_networks_test.py ....
../../../../../docker-py/tests/integration/models_nodes_test.py .
../../../../../docker-py/tests/integration/models_resources_test.py .
../../../../../docker-py/tests/integration/models_services_test.py ....s
../../../../../docker-py/tests/integration/models_swarm_test.py .
../../../../../docker-py/tests/integration/models_volumes_test.py ..
../../../../../docker-py/tests/integration/regression_test.py ......

Could this be just a glitch, has anybody seen it before ?

aboch · 2017-03-14T18:37:17Z

All green.
ping @aaronlehmann @mavenugo @cpuguy83 @thaJeztah @vdemeester

aaronlehmann · 2017-03-14T18:58:53Z

api/types/swarm/network.go

@@ -82,6 +82,7 @@ type NetworkSpec struct {
 	IPv6Enabled         bool         `json:",omitempty"`
 	Internal            bool         `json:",omitempty"`
 	Attachable          bool         `json:",omitempty"`
+	Ingress             bool         `json:",omitempty"`


Add to swagger.yaml

aaronlehmann · 2017-03-14T18:59:19Z

api/types/types.go

@@ -400,6 +400,7 @@ type NetworkResource struct {
 	IPAM       network.IPAM                   // IPAM is the network's IP Address Management
 	Internal   bool                           // Internal represents if the network is used internal only
 	Attachable bool                           // Attachable represents if the global scope is manually attachable by regular containers from workers in swarm mode.
+	Ingress    bool                           // Ingress indicates the network is providing the routing-mesh for the swarm cluster.


Add to swagger.yaml

Thanks, I will, along to the addition to the command reference (forgot to add the diff to the commit)

aaronlehmann · 2017-03-14T18:59:22Z

api/types/types.go

@@ -431,6 +432,7 @@ type NetworkCreate struct {
 	IPAM           *network.IPAM
 	Internal       bool
 	Attachable     bool
+	Ingress        bool


Add to swagger.yaml

aaronlehmann · 2017-03-14T18:59:42Z

cli/command/network/create.go

@@ -59,6 +60,8 @@ func newCreateCommand(dockerCli *command.DockerCli) *cobra.Command {
 	flags.BoolVar(&opts.ipv6, "ipv6", false, "Enable IPv6 networking")
 	flags.BoolVar(&opts.attachable, "attachable", false, "Enable manual container attachment")
 	flags.SetAnnotation("attachable", "version", []string{"1.25"})
+	flags.BoolVar(&opts.ingress, "ingress", false, "Swarm routing-mesh network")


"Create a swarm routing-mesh network"?

aaronlehmann · 2017-03-14T19:04:27Z

daemon/network.go

+	if ingressWorkerStop != nil {
+		close(ingressWorkerStop)
+		ingressWorkerOnce = sync.Once{}
+		ingressID = ""


Without locking, this can race with the goroutine in setupIngressWorker.

aaronlehmann · 2017-03-14T19:06:10Z

daemon/network.go

+func (daemon *Daemon) stopIngressWorker() {
+	if ingressWorkerStop != nil {
+		close(ingressWorkerStop)
+		ingressWorkerOnce = sync.Once{}


What's the purpose of resetting ingressWorkerOnce?

When the nodes leaves the swarm, and joins a swarm again.

Is there an external guarantee that stopIngressWorker can't be called concurrently with SetupIngress or ReleaseIngress?

I think I'd feel more comfortable having a mutex and a boolean instead of a sync.Once. Then the mutex could also protect ingressID, and we wouldn't have to make assumptions about which functions can be called at the same time.

Is there an external guarantee that stopIngressWorker can't be called concurrently with SetupIngress or ReleaseIngress?

Yes, stopIngressWorker is called when docker leaves the swarm.
SetupIngress and ReleaseIngress cannot be called concurrently as they are all queued.
The worker logic is in fact to serialize the setup/release events.

I was under the impression setup or release which in turn are called by executor.Configure() cannot happen to be called after DaemonLeavesCluster().

Given the worker drains ingress network creation/deletion events, I did not expect a long list of events left in the queue to process.

Ok, let me think about it

Actually this can be resolved by moving the ingressID reset into the setupIngressWorker() go routine when reacting to the stop event: L122.

Because only one select at the time will run, and the one that executes setup/release function will read ingressID value and pass it to the functions.

What confuses me about ingressWorkerOnce is that the use of sync.Once implies there's synchronization involved, but the fact that we're resetting it without a lock implies synchronization isn't an issue. Reviewing the places where ReleaseIngress and SetupIngress are called, I don't think they can be called simultaneously. If this is the case, I think it would be clearer to just use a boolean and an if statement. Otherwise, the code looks suspiciously like incorrect concurrent code.

The reason for using once here is not because we are in a concurrent prone code path. It is simply for assuring we initialize the worker once, given we do lazy initialization of the worker.
Therefore only the first call to SetupIngress from the executor will result in starting the worker.

aboch · 2017-03-14T22:45:13Z

@aaronlehmann Addressed your comments. PTAL.

aaronlehmann · 2017-03-14T23:17:38Z

daemon/network.go

+)
+
+func (daemon *Daemon) setupIngressWorker() {
+	ingressWorkerStop = make(chan struct{}, 1)


This channel doesn't need to be buffered

Will change

aaronlehmann · 2017-03-14T23:27:25Z

daemon/network.go

-	ingressChan <- struct{}{}
-	return func() { <-ingressChan }
+func (daemon *Daemon) stopIngressWorker() {
+	if ingressWorkerStop != nil {


This if statement also strikes me as suspicious. It implies that stopIngressWorker can be called before setupIngressWorker. But if that's the case, it suggests that after leaving a swarm and rejoining, we could close an already-closed channel (ingressWorkerStop never gets reset to nil).

Again, I think adding locking around this stuff would make it a lot easier to be confident that there aren't concurrency bugs. If nothing else, it would make my review a lot easier.

It implies that stopIngressWorker can be called before setupIngressWorker

I see your point.

The fact is the SetupIngressNetwork being called from the executor and the call to DaemonLeavesCluster happen from two different threads. Their execution order is not guaranteed, and I saw it not deterministic during the testing of initial diffs.

In fact in the setup ingress call, we need to wait for agentInitWait() for this reason.

So it is not excluded a swarm join quickly followed by a swarm leave would flip the order of events.

So, on DaemonLeavesCluster, I cannot assume the worker was started.

If locking this section ease the code reading, I agree we should do it. I will update shortly.

But this should not be confused with the fact I use the once, which are being used for lazy initialization of the worker as I explained in the other comment.

aboch · 2017-03-15T01:51:49Z

Updated.

aaronlehmann · 2017-03-15T02:42:51Z

daemon/network.go

+	ingressJobsChannel chan *ingressJob
+	ingressWorkerStop  chan struct{}
+	ingressID          string
+	ingress            sync.Mutex


Thanks for adding the lock. But startIngressWorker is using ingressWorkerStop without holding the lock, and SetupIngress / ReleaseIngress are using ingressWorkerOnce without holding the lock, so unfortunately my concerns about concurrency are not fully addressed.

Earlier in the thread you said:

The fact is the SetupIngressNetwork being called from the executor and the
call to DaemonLeavesCluster happen from two different threads. Their execution
order is not guaranteed, and I saw it not deterministic during the testing of
initial diffs.

In fact in the setup ingress call, we need to wait for agentInitWait() for
this reason.

So it is not excluded a swarm join quickly followed by a swarm leave would
flip the order of events.

...so I still don't see how it's safe to use these variables from different goroutines without a lock.

SetupIngress / ReleaseIngress are using ingressWorkerOnce

Not sure I follow. Once.Do(f) is designed to not allow more than one execution of f, even in concurrent environment. I do not think there is any need to hold a lock during a call to Once.Do.

Being called in a once do construct, startIngressWorker is guaranteed to be called once by one thread.

Still, as you said, startIngressWorker initializes the two channels, and stopIngressNetwork, which also accesses the channels, could be concurrently called. To protect against this, both startIngressWorker and stopIngressNetwork are now locking around the access.

Yes, then it spawns the goroutine which listen on those channels, with no protection.

The correct thing here is probably to define a type ingressWorker struct with start, stop, enqueue methods, so that it can control the access to and lifecycle of its channels.

I do not think there is any need to hold a lock during a call to Once.Do.

There is if you may overwrite the sync.Once from another goroutine.

Oh ok, you are referring to the reset of the once variable.
Yes, that is a problem.

The more I look at this, it seems to me the best tradeoff now is to not to stop the goroutine once it is started.

@aaronlehmann I decided to fall back to not stopping the routine, when the node leaves the swarm.

IMO it is fine for now as we are adding the new functionality.
We can later improve the logic getting to ideal stop of the routine, with calm.

I updated the diffs and tested the add/rm ingress, before and after leaving and rejoining the cluster.

aaronlehmann · 2017-03-15T02:44:39Z

daemon/network.go

+	ingress.Lock()
+	if ingressWorkerStop != nil {
+		close(ingressWorkerStop)
+		ingressWorkerStop = nil


This pattern may cause the goroutine in startIngressWorker to miss the channel close, because it might see the nil channel instead of the one that was closed. Also, the race detector will complain (and this might cause problems, depending on the internal implementation of select).

That's right. I added this in last diff round and this is wrong and not needed.
The existing check if ingressWorkerStop != nil is just to tell us whether a worker routine is actually running, if so signal the routine to stop.

Signed-off-by: Alessandro Boch <[email protected]>

mavenugo · 2017-03-25T02:23:36Z

@aaronlehmann I understand that this PR is blocking other Swarmkit vendoring PRs. I am waiting for @aboch to be back so that he can respond to the compatibility question.

aboch · 2017-03-25T04:13:11Z

@mavenugo
I think proper documentation in the update section will need to be added.
So that user will not attempt the ingress nw rm on a mixed version cluster.

Although I think it is very unlikely admin would remove the routing-mesh on a half upgraded prod environment, it is true casual user may hit the problem, so I am proposing we can follow the same approach used in docker network prune, where the cli client prints a clear warning and asks for user confirmation.
Something like

$ docker network remove ingress
WARNING! You are removing the routing-mesh network. Proceed only if all nodes in the cluster are at version 17.05.
Are you sure you want to continue? [y/N]

Higher level mgmt tools can already reject the request given they likely have notion of the engine version running on each node.

mavenugo · 2017-03-25T13:41:17Z

@aboch I agree. if we can do this change, I think we will have the necessary UX covered and also when the proper capability exchange mechanism arrives, it can be automated as well.

Can you pls take care of this change and we can get this merged.

aboch · 2017-03-26T14:16:05Z

@mavenugo Updated.

vdemeester · 2017-03-26T18:01:36Z

@aboch build failure that seems legit

15:08:31 ----------------------------------------------------------------------
15:08:31 FAIL: docker_cli_swarm_test.go:416: DockerSwarmSuite.TestSwarmIngressNetwork
15:08:31 
15:08:31 [df0c8073844d8] waiting for daemon to start
15:08:31 [df0c8073844d8] daemon started
15:08:31 
15:08:31 docker_cli_swarm_test.go:425:
15:08:31     c.Assert(err, checker.IsNil, check.Commentf(out))
15:08:31 ... value *exec.ExitError = &exec.ExitError{ProcessState:(*os.ProcessState)(0xc42020fb40), Stderr:[]uint8(nil)} ("exit status 1")
15:08:31 ... Error response from daemon: rpc error: code = 6 desc = ingress network (pmoutasdajym5jdq9jxs1zlx7) is already present
15:08:31 
15:08:31 
15:08:31 [df0c8073844d8] exiting daemon
15:08:34 
15:08:34 ----------------------------------------------------------------------

aboch · 2017-03-26T19:45:21Z

Thanks @vdemeester
Now that I changed the cli to ask for confirmation, and not having a force option for docker network remove, I'd better change the respective integ test to use the API calls directly.

Signed-off-by: Alessandro Boch <[email protected]>

mavenugo · 2017-03-27T03:10:13Z

@aboch I think it is better to introduce the -f for docker network rm and handle it for all other regular remove case as well. This will also force the user to use this flag when removing the ingress network. But that is beyond the scope of this PR and hence am good with the current PR.

vdemeester · 2017-03-27T07:45:55Z

@mavenugo we can add the -f/--force flag in another PR (follow-up). I can take care of that 😉

vdemeester

LGTM 🐯

mavenugo · 2017-03-27T14:30:39Z

Thanks @aboch @vdemeester . LGTM.

Allow user to replace ingress network

edesai · 2017-06-28T18:22:19Z

I am using 17.05 for my manager and worker nodes and am trying to create a new ingress network with custom subnet and then creating swarm service. But service doesn't seem to spawn any containers. I see some errors from docker logs. Seems like it keeps "looping over" Requesting Address from the new subnet. Can someone help out here?

Logs as below:

Jun 28 10:58:24 dockerd[19967]: time="2017-06-28T10:58:24.952553135-07:00" level=debug msg="Service xa3gd5xvpgatbwwhis1of9lp2 was scaled up from 0 to 1 instances" module=node node.id=b9zadh96hjod5zrx55lvsv19m
Jun 28 10:58:24 dockerd[19967]: time="2017-06-28T10:58:24.953912764-07:00" level=error msg="task allocation failure" error="service xa3gd5xvpgatbwwhis1of9lp2 to which this task lukp3bld3w35g560h5ufr6ae0 belongs has pending allocations" module=node node.id=b9zadh96hjod5zrx55lvsv19m
Jun 28 10:58:24 edesai-redhat-7-3-dcnm dockerd[19967]: time="2017-06-28T10:58:24.953963436-07:00" level=debug msg="RequestAddress(GlobalDefault/11.11.0.0/16, , map[])"
Jun 28 10:58:24 edesai-redhat-7-3-dcnm dockerd[19967]: time="2017-06-28T10:58:24.954710522-07:00" level=debug msg="RequestAddress(GlobalDefault/11.11.0.0/16, , map[])"
.
.
.

My steps of reproducing are:

docker network rm ingress
docker network create --driver overlay --ingress --subnet 11.11.11.1/16 --gateway 11.11.11.1 my-ingress
ikhelr9jnjllg8fr5xfk9p8bg
docker service create --network my-ingress ct:0.1
xa3gd5xvpgatbwwhis1of9lp2
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.

Observations:

docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
xa3gd5xvpgat condescending_spence replicated 0/1 172.28.x.x:5000/ct:0.1
[root@edesai-redhat-7-3-dcnm ~]# docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
xa3gd5xvpgat condescending_spence replicated 0/1 172.28.x.x:5000/ct:0.1

docker network ls
NETWORK ID NAME DRIVER SCOPE
e88846664160 bridge bridge local
fde9fb9ec622 docker_gwbridge bridge local
b3efd1b6e7f4 host host local
ikhelr9jnjll my-ingress overlay swarm
5e8e58b59631 none null local

docker network inspect ikhelr9jnjll
[
{
"Name": "my-ingress",
"Id": "ikhelr9jnjllg8fr5xfk9p8bg",
"Created": "2017-06-28T10:57:37.24037921-07:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "11.11.11.1/16",
"Gateway": "11.11.11.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": true,
"Containers": {
"ingress-sbox": {
"Name": "ingress-endpoint",
"EndpointID": "940a99930934e923499d9ba59e87e166f5c1333a7010411f0968a1335d43bb59",
"MacAddress": "02:42:0b:0b:00:02",
"IPv4Address": "11.11.0.2/16",
"IPv6Address": ""
}
},

thaJeztah · 2017-06-29T00:38:29Z

@edesai please open an issue instead of commenting on a closed pull request

Also keep in mind that the GitHub issue tracker is not intended as a general support forum,
but for reporting bugs and feature requests. For other type of questions, consider using one of;

the Docker Support Forums - https://forums.docker.com
the Docker community Slack channel (register here: http://dockr.ly/community)
StackOverflow

realcbb · 2017-10-13T06:19:57Z

@thaJeztah Can driver of ingress network be customized, or can port 4789 UDP of overlay driver be changed? In my scenario, 4789 UDP is already used by the underlying network of VMs which cannot be modified.

When I use weave network plugin v2 for swarm, communication between containers on different hosts works well, but it seems that the network is still blocked when routing mesh(LB) happens.

GordonTheTurtle added the status/0-triage label Mar 9, 2017

aboch mentioned this pull request Mar 9, 2017

Allow replacing ingress network moby/swarmkit#2028

Merged

vdemeester added status/1-design-review status/needs-vendoring and removed status/0-triage labels Mar 13, 2017

aboch mentioned this pull request Mar 14, 2017

Vendor swarmkit 9fdea50 #31808

Closed

aboch added this to the 17.04.0 milestone Mar 14, 2017

aboch added rebuild/janky and removed status/needs-vendoring labels Mar 14, 2017

GordonTheTurtle removed the rebuild/janky label Mar 14, 2017

aaronlehmann reviewed Mar 14, 2017

View reviewed changes

aaronlehmann reviewed Mar 15, 2017

View reviewed changes

aboch added 2 commits March 24, 2017 11:07

Vendoring swarmkit @9fdea50

434e969

Signed-off-by: Alessandro Boch <[email protected]>

Vendor libnetwork @b6cb1ee

ff049a4

Signed-off-by: Alessandro Boch <[email protected]>

Allow user to modify ingress network

d59d19c

Signed-off-by: Alessandro Boch <[email protected]>

aboch added the rebuild/experimental label Mar 27, 2017

GordonTheTurtle removed the rebuild/experimental label Mar 27, 2017

aboch added the rebuild/experimental label Mar 27, 2017

GordonTheTurtle removed the rebuild/experimental label Mar 27, 2017

vdemeester approved these changes Mar 27, 2017

View reviewed changes

vdemeester mentioned this pull request Mar 27, 2017

Add support for --force in network rm #32125

Closed

mavenugo merged commit 04295d2 into moby:master Mar 27, 2017

dongluochen mentioned this pull request Mar 29, 2017

Swarm has difference in network inspect from docker docker-archive/classicswarm#2666

Closed

mavenugo mentioned this pull request Mar 29, 2017

SSH over OpenVPN connection fails when Docker Swarm is running #31972

Closed

dnephin pushed a commit to dnephin/docker that referenced this pull request Apr 17, 2017

Merge pull request moby#31714 from aboch/cingr

24faf9e

Allow user to replace ingress network

mdlinville mentioned this pull request Apr 21, 2017

Document customizing the ingress network docker/docs#2998

Closed

mavenugo mentioned this pull request Jun 12, 2017

Docker stack deploy cannot deploy service to different node in swarm cluster, always report that ingress network overlap #33626

Closed

albers mentioned this pull request Jun 24, 2017

Add bash completion for docker network create --ingress docker/cli#239

Merged

aboch deleted the cingr branch November 8, 2017 19:52

thaJeztah mentioned this pull request Jul 6, 2022

Cannot set sysctl's in a host network namespace without --privileged #43769

Closed

Allow user to replace ingress network #31714

Allow user to replace ingress network #31714

Conversation

aboch commented Mar 9, 2017 • edited Loading

vdemeester commented Mar 13, 2017

thaJeztah commented Mar 13, 2017

aboch commented Mar 13, 2017

mavenugo commented Mar 13, 2017 • edited Loading

aboch commented Mar 13, 2017 • edited Loading

aboch commented Mar 13, 2017 • edited Loading

aboch commented Mar 14, 2017 • edited Loading

aboch commented Mar 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aboch commented Mar 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aboch Mar 15, 2017 • edited Loading

Choose a reason for hiding this comment

aboch commented Mar 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mavenugo commented Mar 25, 2017

aboch commented Mar 25, 2017

mavenugo commented Mar 25, 2017

aboch commented Mar 26, 2017

vdemeester commented Mar 26, 2017

aboch commented Mar 26, 2017

mavenugo commented Mar 27, 2017

vdemeester commented Mar 27, 2017

vdemeester left a comment

Choose a reason for hiding this comment

mavenugo commented Mar 27, 2017

edesai commented Jun 28, 2017

thaJeztah commented Jun 29, 2017

realcbb commented Oct 13, 2017

aboch commented Mar 9, 2017 •

edited

Loading

mavenugo commented Mar 13, 2017 •

edited

Loading

aboch commented Mar 13, 2017 •

edited

Loading

aboch commented Mar 13, 2017 •

edited

Loading

aboch commented Mar 14, 2017 •

edited

Loading

aboch Mar 15, 2017 •

edited

Loading