##Why Benchmarks are used and the pitfalls of benchmarks
-
Why are benchmarks used? Benchmarks are used for good reasons and a multiple bad reasons. Benchmarks are used simply to measure the performance of multiple processors. Benchmarks are a go to for performance for a multitude of reasons. First the available information on a CPU is extremely limited on a store listing or even on the box itself. Second the way the benchmarking program is a great way to see how well a processor performs in real life not just a lab. Finally benchmarks are a good way of seeing how much value you are getting for the price you pay. Benchmarks generally include the name of the processor, the score the processor receives, and the cost of the processor. Unfortunately even with all of these good things benchmarks are able to provide, benchmarks suffer from some of the worst pitfalls imaginable.
-
The first and major problem of benchmarks is that for processors the code they run to test is not always optimized for those specific types or architecture. While that argument might sound silly it doesn’t effectively test the hardware. Imagine that a benchmarking program that isn’t optimized for any specific hardware, that benchmarking software isn’t making use of the hardware’s potential ability. Rather it is testing how well code can be pushed through the hardware. This is not an accurate measure of the processor because one benchmark might say that an AMD processor is better then an Intel processor but with optimization the Intel processor can out preform the AMD processor when both have proper optimization.
-
The next problem with benchmarks, more so for CPU benchmarks, is the use of multiple cores. This is important for AMD processors as generally they have up to 8 physical cores to process information. This will help the computer to multitask by dividing up the instruction sets to the different cores of the CPU. The biggest problem is that most benchmarks do not test multitasking capabilities. Most benchmarking software really can only test 2 cores effectively in order to be fair to a majority of CPUs. This has started to shift in a more positive direction with some programs being able to test up to 4 cores effectively. With the newest Intel chip actually having 8 physical cores along with AMD already having a long line of 8 cores already, it can be assumed that the benchmarking software will then shift to being able to test more than 2 to 4 cores.
-
Another problem that benchmarks do not test is the disparity between operating systems. While the benchmarking programs have to worry about optimization for hardware they also have to worry about optimization for the operating system. The way a program sends instructions to the CPU also depends on the operating system. This can make a certain program run slower of faster depending on how well optimized it is for that specific operating system. An example of this is how well Itunes runs on windows vs. an apple computer. Itunes is developed by apple and thus will be optimized for an apple computer. Windows can run Itunes but it does not run it well. This is not because the hardware on a mac is better rather that the program itself is optimized to run better on an apple computer than a computer that runs windows.
-
The most problematic thing about benchmarks is that rather than telling you what the processor is mostly used for, the price of that processor, and the power consumption of that processor; they put out an arbitrary number that is a comparison point for the different processors. This is, however, better on graphics cards because the comparison is generally on frames per second but still some benchmarks use an arbitrary score. While the benchmarks will show the difference between the two processors in terms of processing power, minus the problems listed beforehand, does not give a measurable difference in numbers. What is the difference between a processor with a result of 9,000 and another one with 11,000? The simple answer is 2,000 but what does that mean in terms of processing power, power consumption, and multitasking? Even more so what is the difference between one that scores 7,000 and one that scores 6,996? Does the difference of 4 even matter with such high numbers?
-
Unfortunately even with all the problems of benchmarks most people still use them as an entry point of determining what processor to buy. The reason why even people who know of the pitfalls still use benchmarks is because of the lack of information available about a processor when buying one. Looking at a newegg store page for a AMD processor, the information available are: the number of cores, the operating frequency, the L2 cache, L3 cache, and the power consumption. No where in the information provided is the CPI listed. How you determine the performance of a processor is the execution time where the instruction set is the same between the two processors being compared. The execution time of a given processor is determined by the number of instructions multiplied by the CPI; then that number is divided by the clock rate of the processor. This is why the clock rate is not the only determining factor when comparing processors. So without companies able to release a CPI number and the corresponding instruction set, benchmarks are a way to get a gauge of the performance of a given processor.
-
The best way to fix the problems with benchmarks are to show the execution time rather than the arbitrary number that is produced. Show the CPI of the processor along with the instruction set as well. How to fix the multi-core benchmarking problem honestly cant be solved simply. Sure you can find the CPI and then find the execution time bu that wont account for how well the processor will do running multiple programs. The fix to this is simply a choice between what type of processor you are looking for and what kinds or programs you are going to be running. Intel has a reputation for being the go to processor for gaming and that holds true for the most part. Games make use of up to 2 cores on a given processor, meaning if your processor only has 2 physical core and 2 virtual cores then yes games will run better on that machine. In contrast if you are trying to render videos, compile code, or any task that will run multiple programs at once then an AMD processor which in general has 4 physical cores then you should look to getting a AMD processor. Laptops have a limited power supply meaning that if you are trying to expand the up time of it then you should lower the power consumption. This means the power consumption of your CPU will make a huge impact on the lifespan of your battery. In this case numbers are easy to look at and see that Intel has a lower power consumption rate than an AMD processor. If money is an issue then you want the best price per performance. AMD typically is cheaper than Intel while still competing in the performance department. All of these factors matter when shopping and choosing the right processor for you.