Vertexbuffer.com

Without a benchmark, there may be no way to find out if we have improved our performance. When we make an optimization that improves performance from 10fps to 60fps it is obvious; but could you visually determine a framerate increase from 60fps to 120fps without a frame counter? Probably not.

What is a benchmark? The word benchmark was originally a surveying term. When a builder begins to layout his plans from paper to the actual site, he needs a common fixed point that won’t move. Then all of the other measurements can be relative to this point. In other words, the benchmark is a starting point for comparison.

A benchmark is the starting point for comparing what we had to what we have; and in a video game, benchmarking is quite different from that of other types of programs. Examine these guidelines for determining a benchmark:

• Repeatable
• Representative
• Easy to Run

Let’s dissect these traits and compare them with our ability to set a benchmark using these characteristics.

Repeatable. Hrm… games aren’t always deterministic. That is, if you run the exact same input, you are not always guaranteed the same output. This is due to random number implementations within many of our game subsystems. AI and particles are two systems that may not behave exactly the same every time. Even the laws of physics have an element of randomness when dependent on updates across a network. So games aren’t very repeatable. Strike one.

Representative. Hrm… this one is an issue as well. The representative user of a game is a pc gamer. We can try to represent them, but that’s difficult. Writing AI that behaves like we do is very difficult. Our platform, at least on the PC will very from machine to machine. In fact, in a set of 30 similar machines, in terms of hardware, may vary greatly due to software differences. Strike two.

Easy to run. Hrm… Yea, we can do this! Many games engines are driven by a console and/or scripting language. We can create test scenes using this language to get as close as we can to a representative and repeatable test. So all hope is not lost.

Instead of suggesting that these are the required attributes of a good benchmark, we must think of them as our goals to achieving the best benchmark.
There is one more piece we are missing. What are we measuring? In video games that one is easy. FPS! Frames per second, or sometimes called Hertz, meaning number of times per second, is the de-facto measure for performance in video games. If you increase the framerate, you can add more content or runtime processing. If you decrease framerate, than you are limiting the ability to add content or runtime processing.