Changes / Updates for Upcoming Round #4630
Comments
|
Any estimation when you will start the round? |
|
We'd like to pull the next round from continuous runs this week. Looks like there was an outage at the office this weekend; we'll get things back online Monday morning. |
|
@nbrady-techempower Thanks. Can you please merge #4671 so it makes it into the next round? Thanks! |
|
This probably got lost before, but I found another difference between the Terraform scripts and the Azure portal defaults - the OS disk type is "Standard HDD" instead of "Standard SSD". Are you going to change that too in the upcoming Azure runs? |
|
@volyrique I want to point out that we're not necessarily shooting for the most optimized environment for azure; I think some of the point to the azure tests is to see how tests perform in a less optimized environment. With that being said, I believe we're using the temp storage drive in azure which is SSD (https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual-machines-windows-sql-performance#temporary-disk) I'll make sure to get the final specs posted soon. |
|
seems tfb is stuck |
I checked the logs - it looks like it's running fine to me. |
|
maybe something with cache? |
|
It has only been running for 2 hours. It is still going and I see no reason to think anything is working improperly based on the logs and expected duration. |
|
It does appear to be stuck; we'll take a look at it. |
@nbrady-techempower Sure, my main point was about reproducibility - I don't mind if a particular Azure option that may improve performance is not used, as long as it is clearly documented (especially if it is not the same as the default), so that people who try to reproduce the results are not left wondering why their data differs so much. |
|
I've added some more info on the wiki https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Environment Once I finalized the azure terraform setups and the scripts I'll make a reference of that on the wiki as well, since all the details can be found there. |
|
Seems Azure is stuck! |
|
We may be pushing this off for just a little bit until we can resolve the Azure issues. Hopefully this week. |
|
Still working on Azure. We're hoping to have Round 18 published by the end of next week, barring any more unforeseen issues (there are always some!) Feel free to continue making adjustments, but I wouldn't make any experimental changes as they may get captured in official round results. |
|
Another note regarding Java versions - most images are using the Docker |
|
It seems that the CVE fixes had a pretty significant effect on the JSON serialization, single query, and fortunes results in the latest Citrine run. I don't think the framework standings changed much, but the maximum requests per second dropped by 22.04%, 16.75%, and 22.05% respectively. |
|
Another important point about the MDS mitigations - it seems that the results are with activated hyper-threading.
Search for the text above here. The Xeon Gold 5120 is having 14 physical cores. I suspect there will be a run with hyper-threading disabled. |
|
Apparently other projects suffered from regressions as well. |
Yikes. |
|
In the plaintext bench, the difference is near to 50% in some frameworks/platforms.
I think like @zloster : run with hyper-threading disabled. |
|
If everything goes as planned, Round 18 will be published next week (approximately July 8). It will be based on the most recent Azure run and the Citrine run that used the same commit. These were captured prior to application of the CVE-2019-1147x mitigations. |
|
@bhauer those runs still contain "onyx" framework, which violates db bench requirenments |
|
@fafhrd91 See the original post. Anything that's listed as violating the rules won't be displayed on the official results page. |
|
For those just following along here, Round 18 has been released. https://www.techempower.com/blog/2019/07/09/framework-benchmarks-round-18/ |
|
@nbrady-techempower Thanks for tracking this for the round. Should we close this one and start a similar tracking for Round 19? |
|
@bhauer Absolutely; will get one set up. |
I'll be editing this pinned issue to link to other issues or pull requests that may affect tests for this round.
We are working toward Round 18 now. Official results have been captured and will be available on the results website soon.
#4291 - Responses including cached date header
#4493 - Limit the amount of memory the framework container can use
#4601 - Cached queries tests were using the wrong levels
#4622 - Stricter Multiple Query test verification
#4636 - Enable accelerated networking in Azure
#4642 - PostgreSQL bumped to v11
#4834 - Actix fortunes explanation of results
We've also clarified some rules:
Tests that were in violation of the rules when captured:
The text was updated successfully, but these errors were encountered: