Test 5: Updates

[bhauer]

See issue #133 concerning additional test types.

We are working on a test to exercise database updates. This is a preliminary specification and we may iterate this some. I've created this issue because we've already received an implementation of Test 5 prior to any specification being written up. If you want to begin implementation prior to a final specification, please just recognize that we may change the specification.

Test 5 is likely to be a variation of Test 3, the multiple database query test. Analogous to Test 3, it will include a URL-tunable parameter ("queries") specifying the number of updates to run.

• The "queries" parameter must be bounded by the application to between 1 and 500. Any non-integer parameter should be treated as 1. Any integer less than 1 should be treated as 1. Any integer greater than 500 should be treated as 500.
• The "queries" parameter specifies the count (N) of rows that require updating.
• As with Test 3 (the multiple queries test), this test operates on the Worlds table that is known to have 10,000 rows.
• For each update, the implementation should use the ORM to fetch a random row from the Worlds table, update the RandomNumber field to a new random integer between 1 and 10000, and then persist the updated row.
• For each update, the resulting in-memory World object reflecting the new random number should be added to a list.
• The resulting list of N updated rows should be returned as a JSON response. The updated values for the RandomNumber field should be reflected in the JSON response.

If available, the implementation may use a batch update statement. The implementation may use transactions.

For raw tests (that is, tests without an ORM), we still expect each updated row to receive a unique new RandomNumber value. It is not acceptable to change the RandomNumber value of all N rows to the same random number using an UPDATE ... WHERE id IN (...) clause.

[bhauer]

Some important notes for test 5:

• Our i7 hardware does not use SSDs and therefore the i7 results will be very poor, with strong convergence across frameworks at 20 updates per request. We would like to eventually run this test using SSD-equipped hardware, but we do not have that at our disposal presently.
• Although in our spot testing, the MySQL ISAM storage engine is significantly faster with writes, even on our hardware's plain magnetic disks, and therefore demonstrates a broad convergence across frameworks, we will continue to test using InnoDB. ISAM is not conventionally considered the production-class storage engine option.

[bhauer]

The requirements for Test 5 are now ready for use and available here: http://www.techempower.com/benchmarks/#section=code

[Skamander]

Just out of curiosity: why did you choose to send the server header? Isn't it good practice, or at least encouraged [1] to deactivate it?

[1] - http://www.troyhunt.com/2012/02/shhh-dont-let-your-response-headers.html

[bhauer]

Hi @Skamander. I'm not opposed to deactivating it, but several of the tests are sending a Server header and I think a majority of sites still do the same. Making it required is part of an effort to make the number of headers and rough payload size of the response slightly more normal across frameworks.

[pfalls1]

@Skamander my preference would be to try and remove the Server header as much as possible. While it's possible that we'll only be able to achieve this to a certain degree, the servers that end up sending some sort of Server header will only be penalized by the added payload of the response, which in my opinion is a good thing since they aren't fully conforming to the test parameters.

[Skamander]

Hi @bhauer, @pfalls1 - thanks for your response. I looked at the response headers of amazon, google, facebook, twitter, pinterest and a few other big sites earlier, and surprisingly only facebook deactivates the server header. google, twitter and amazon send a custom server header (google for example gws). Not sure if it really is best practice like I thought (and many blog posts suggest). ^^

[bhauer]

Closing this issue since we've got Test 5's requirements solidified now.