Your ideal benchmark?

[GNEP]

Given that we should all accept that whatever happens, benchmarks will be used, how would you do it?

I have just done a mock-up result attached. If it were possible, I would "record" a typical level in, say, UT2003, at 1280x1024 (for argument's sake), with the max aniso and AA settings that a card allows (I like purdy things...).

Then use a utility like FRAPS (although even this can't do what I want) to record into a log file the effective FPS every, say, second, and then chart them against time.

Ideally if I was a reviewer I would have a timeline of the level showing screenshots of what is going on at key points (eg where there is most stutter).

Rinse and repeat for other cards in the exact same machine with the exact same "recording" of the level.

IF we use benchmarks, this is how it should be done.

See the attached.

[GNEP]

And this is how some sites might represent the exact same results (attached).

Which card is best for you?

What other methods would there be in a slightly more ideal world?

[GNEP]

Hmmm... someone else was thinking the same as me...

http://www.hardware.fr/art/lire/430/4/

[Dogbert]

Hmm, according to your graph, card 2 seems the coolest in my opinion since it has the most stable framerate.

The usual bar chart hides this (very) important information.

I pretty much agree with the way of testing showed on the 1st post.


There are some big issues about benchmarks though. Benchmarks usually involve the game, cpu and graphic card while in the real world it'll also include sound, network I/O and input devices.
This means less cpu dedicated for graphics, which could mean very very different results.

Because of this, I'd use the timeline charts comming from a 'lan party' of identical computers with different graphic cards.
I'll let the players play a few games in the same map switching from one computer to the other till everybody played on all the computers.
After every game I'd ask the players for their input about picture quality, sound and overall responsiveness.

[GNEP]

Yup, I'd agree that subjective tests are the best when talking about game "feel"... however the initial assumption that I made was that people want to see benchmarks and graphs of some sort.

And LAN parties etc would be a good place to do this - I think that Rags mentioned in the "in the house" thread that he had done this without telling people that there was a P in one of the machines. And the P machine everyone agreed felt "smoother" than the others. (iirc, he also had a GF4Ti4600 and possible an R8500 in there somewhere.).

But for the ideal objective test, where do we go?

1) Hold all other conditions the same (i.e. machine speed, network speed etc)

2) Use a real, recent game (or lots of them)

3) Set some sort of IQ level requirement - it doesn't all have to be the same res, but should be the same effective quality - this will inevitably be the most subjective part.

4) Try to measure "fun" levels or perhaps "lack of frustration" levels - by graphing as per above, or maybe just minimum FPS, or perhaps modelling with a (normal, poisson, whatever) statistical mindset, and showing the results as mean and s.d.

5) Add subjective qualifiers. Lots of them.

What else?

gnep

[Dogbert]

You should add stability tests, run all computers at the most demanding settings overnight.

Check card stability over non standard (OC) AGP speeds.

Run some 2D benchmarks as well.

And last but not least, check multimedia capabilities and quality:

* TV-out
* DVD Playback quality
* Multi monitor support features and quality

Very important (mostly checked for sound cards and not for graphic cards): cpu usage during normal operation with multimedia features enabled. (for example, run a DVD movie on your tv-out).

[GNEP]

true, true. But I think that any non games-only reveiws are doing a lot of this already. It is the standard (and volume) of stupid FPS benchmark graphs that worries me most in the currect crop of reviews. I am just looking for alternatives to those.

Regarding non-3d games, I don't think that anyone could really argue that the Parhelia is the best card going (I think) - but stability over prolonged periods, with the drivers on the shipped CD - yes, that is exceedingly important.

As is checking compatability/conflicts with a wide range of other hardware - how does it work on an ECS K7AMA with an Audigy for example.

gnep

[Dogbert]

Actually I'd add another benchmark, one that (sadly) doesn't exist anywhere.

I'd call it "out of the box experience". No driver download, no patches no nothing. Take all kinds of cards straight out of their boxes and try them.

[GNEP]

I'd say that was the most relevant one for a $100 vid card - perhaps not for a $400 one...

Right.... now we have the benchmarks, who wants to give me a Parhelia, a GF4Ti4600, a good R8500 and a 3dLabs Wildcat to try them on? Anyone?

[Dogbert]

I can lend you my Millenium (I) and a Diamond Voodoo II (8MB)

[GNEP]

Have a Voodoo 3 and a G450 in the computers at home...

It's a start!

Oh - and I need to be sent lots and lots of games

[Pace]

[H]

Where's the "Penis Enlarger Effect" graph?

[GNEP]

Don't need that