You are here: Home > Fastcode project > Ideal Benchmark Documentation

## The Ideal Benchmark

We do not know anything about usage patterns of the function and must assumethat every value of the one and only parameter is used the same number of

times. The benchmark should therefore ideally measure the runtime of the

function with every possible value of the input parameter. Each run should

contribute the same to the overall benchmark.

## The Practical Benchmark

If we call the function with every possible value of the Cardinal inputparameter, the 4G runs would take a very long time. We have to use subsets

of the parameter range. Even numbers are never prime and a given IsPrime

function will be fast when called with an even valued parameter. This way

they do not count much in the benchmark. Because it takes longer time to

verify whether a big number is prime or not compared to a small number, big

parameter values will contribute more to the benchmark if we use an equal

amount of small and big numbers. How the runtime grows with bigger parameter

values depends on the used algorithm in the function. Because many different

algorithms may be used there is no perfect distribution of parameter values.

We only know that there should be more small values than big values.

The Cardinal input range is divided into 3 subranges = 3 subbenchmarks. Each

subbenchmark weigth is adjusted such that the fastest function for each

subbench scores 300 point on my P4 1600. This way functions that perform

badly in a certain subbenchmark is penalized heavily.

The parameter is stepped by this function

Parameter := Parameter * Step + 1

It is coded sligthly different because I use Double for the Step. +1 is to

avoid selecting non primes all the time. If the parameter is a number times

Step then the number is divisible with Step and not prime. The formula

ensures that more small values than bigger values are benchmarked in each

subrange. The use of subranges makes the contribution better.

NumberFP := MINSUBBENCH1NUMBER;

Number := Round(NumberFP);

while Number < MAXSUBBENCH1NUMBER do

begin

Prime := IsPrimeFunction(Number);

NumberFP := NumberFP * SUBBENCH1STEPSIZE;

Number := Round(NumberFP) + 1;

end;

The three subbenchmarks differ in range only.

Shortcomings of the Practical Benchmark

Not all possible values of the parameter is used in the benchmark and it is

possible to optimize the function for these values only and still win with a

function that performs bad on other values. If that happens I twist the

benchmark code a little to use other values and the "winner" drops back.

## Conclusion

It is not possible to create the perfect benchmark and we must live withsomething less. In this case things are pretty simple because the function

only has one input parameter and we can make a pretty good benchmark.