Intermittent 404s during TestScaleTo10

[mattmoor]

e.g. here:

scale_test.go:125: Route "scale-00010-005-quvtpcyb.serving-tests.example.com" prober failed with error: Get unexpected response &{404 Not Found 404 map[Date:[Sun, 09 Dec 2018 18:25:14 GMT] Server:[envoy] Content-Length:[0] Zipkin_trace_id:[734fefe365b6f665e9da511df2a7d3e7]] []}

I suspect that this is caused by the following:

• We have a gateway with N > 1 replicas,
• The new network config rolls out to M < N replicas,
• WaitForEndpointState hits one of the M replicas, and
• The route probes hit one of the N-M replicas

I believe the original workaround for this was this, where we ran a single gateway.

I suspect that we should have this for both gateways like we have here.

However, this workaround breaks down for scale testing, where we may need N replicas to reach a certain scale...

cc @alculquicondor @tcnghia