Throughput problems when compared with Netflix Zuul and Nginx

[turgaycelik]

We performed some tests by using Apache HTTP Server Benchmarking Tool, ab. Results don't seem promising when compared to Netflix Zuul and Ngnix.

Currently, we are performing Gatling tests for Zuul and Ngnix. Since the current performance of Spring Cloud is not promising, we didn't include it as an alternative at least for now.

Experiment	Mean Time Per Request (ms)	Request Per Second
Nginx reverse proxy	32.085	6233.40
Zuul (after warmup)	28.422	7036.90
Spring Cloud Gateway	229.058	873.14

PS: Test server is AWS EC2 M4.2xLarge (Eight-Core CPU, 32 GB of Memory)

[spencergibb]

Can you give any indication of how you configured the gateway, what version you used?

[turgaycelik]

Version:
I checked out the code from master on 22/11/2017 (Revision number 9b61b33).
Gateway Configuration:
I added the following method to GatewaySampleApplication class:

@GetMapping("/ind")
public ResponseEntity<Object> proxyIndex(ProxyExchange<Object> proxy) throws Exception {
		return proxy.uri(home.toString() + "/index.html").get();
}

application.yml

management:
    security:
        enabled: false
remote:
  # address of the web server that hosts the static content.
  home: http://ec2-xx-tt-yyy-zzz.us-west-2.compute.amazonaws.com/

Running the Load Test
I used Apache ab for the load test. The test client is another aws ec2 instance residing in the same region with the server ec2 hosting the spring cloud gateway.
ab -n 10000 -c 200 http://ec2-xx-tt-yyy-zzz.us-west-2.compute.amazonaws.com:8080/ind

Status of the master head
I also checked out master today. I see that there are a lot of changes. I performed equivalent configuration. When I run the test with ab, it throws the following error:
apr_pollset_poll: The timeout specified has expired (70007)
To check whether it is a connection error, I sent a request with curl to the URL from the same test client and it successfully returned the static content.

[spencergibb]

@turgaycelik reactor-netty has issues with http 1.0 and hence ab. reactor/reactor-netty#21

[spencergibb]

I created a simple benchmark project, so I could share what I've done https://github.com/spencergibb/spring-cloud-gateway-bench

SNAPSHOTS have made some changes that I wonder if they had an impact.

I'd appreciate PRs and any updated stats from your tests. I was just using my laptop.

[kentoj]

It looks like we could use h2load to do the testing under HTTP 2.0
See this post.

[rahul-pharande]

Any update here.@turgaycelik could you please share updated performance numbers with latest spring-cloud-gateway?

[VinodKandula]

We had similar performance issues with Spring Cloud Gateway(Finchley.SR1), please find comparison metrics below.

				   Spring Boot 1.5.4 + Zuul Gateway	Spring Boot 2.0.4 + Spring Cloud Gateway(Finchley.SR1)
		
Throughput(Req/Sec)				    460				152
Average Response Time(ms)			    107				323

Test Server Configuration: M4.xLarge AWS Instance — 4 Core CPU, 16GB of Memory

It's very confused state if spring cloud gateway can be used in prod, please provide your comments.

The overall system looks like the following

1. Config Server
2. Discovery Server (Eureka)
3. Zuul/Spring Gateway Server
4. Spring Data (JPA) Rest Repositories(CRUD) Service

All rest endpoints are fired via Zuul/Gateway which uses the discovery server to get the list of available instances.
JMeter is used for Performance tests. It is very straight forward to see the performance results between Zuul vs Sping Cloud Gateway.

#301

[thekalinga]

@spencergibb Do you have a plan to create a project with standardised setup to compare Zull vs Gateway

[spencergibb]

@thekalinga yes, at some point

@VinodKandula Any performance test should be done with the latest boot and Greenwich spring cloud.

[dskaredau]

@VinodKandula can you guys run same perf tests over Greenwich? We are also considering moving from Zuul 1 and now deciding between Zuul 2 or Sping Cloud Gateway

[VinodKandula]

@spencergibb when we have tested, we have used the latest spring boot and spring cloud versions at that time of our testing and since the performance results were not promising when compared to spring cloud zuul, we have moved away from spring cloud gateway. If you are so confident about the Greenwich release Spring cloud gateway team have to show case/publish the performance results and what the changes made to improve the performance.

[VinodKandula]

@demoroz we have moved away from spring cloud gateway to spring cloud zuul. And I don't think spring team is supporting zuul 2

[spencergibb]

@VinodKandula I've published mine #124 (comment)

I'd love to see a project that isn't performant.

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-projects-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.