# Quadrant Benchmark Score Accuracy



## charlie_c (Aug 18, 2011)

So, I wanted to compare benchmarking tools - I'm running (and loving) imnuts TW4 themed EP1W. I used two kernels - the imnuts ext4 kernel included with the ROM and imoseyon's 2.0.7 ext4 voodoo kernel. This test was _not_ about the kernels. I love both, both perform excellently. This comparison is just about how reliable/meaningful the benchmark numbers are.

NOTE - I ran each test twice and took the highest result for each kernel. This wasn't about which kernel scored highest, it was to show consistency from test to test. Everyone's been reporting incredible Quadrant scores so I wanted to see if this was evident on other benchmarks.

First, the usual Quadrant Standard:

View attachment 2511


Clear difference between the two kernels (around 45%!).

Next, I tried Antutu:

View attachment 2512


Basically equal on this, big difference from Quadrant.

Then on to CF Bench:

View attachment 2513


Again, almost completely equal.

Not trying to burst everyone's Quadrant bubbles, but I think, at least based on this, the Quadrant tests for imoseyon's kernel is returning false scores (which I know has been said a million times, I'm hoping this is just evidence). The magnitude of the difference is too much in my opinion, it's only visible in Quadrant, and it's higher than any scores I could hit overclocked at 1.4Ghz on Froyo.

That said, thanks to all the dev's for their work. I think this is the best my phone has ever run (or looked). I can't wait to see what they'll do once the source is released.


----------



## Birthofahero (Aug 18, 2011)

Interesting ....thoughts?


----------



## shrike1978 (Sep 2, 2011)

Birthofahero said:


> Interesting ....thoughts?


The main thing to keep in mind is that benchmarks are artificial. They rarely reflect real world situations. PC graphics card makers take advantage of that a lot. They'll tweak their early drivers to do very well on certain benchmarks rather than focusing soley on real world performance, since reviewers tend to put a lot of weight on benchmarks. It's likely that something in imoseyon's kernel is interacting in a specific way with a specific part of Quadrant and causing a spike in the score. I don't think imoseyon is doing that intentionally. More likely, it is an unintentional side effect of something he's doing.. However, this is a great demonstration of how benchmark numbers can't always be trusted.


----------



## charlie_c (Aug 18, 2011)

Birthofahero said:


> Interesting ....thoughts?


I think benchmarks are meaningless by themselves, their main use is comparing how different devices run a preset series of tests or comparing changes made to a particular phone (overclocking, ext4 conversion, etc). I'm more concerned with the latter. My main concern with the false numbers that Quadrant is giving is that it'll mask the results of any changes. Let's say imoseyon makes a change to the kernel that actually improves speed but the change results in Quadrant suddenly scoring correctly, the score would still drop, giving the impression of a decrease in performance.

People seemed to be getting hung up on the quadrant scores, and if the scores are all false, it's just misinformation and misleading.


----------



## imnuts (Jun 9, 2011)

Just as a note, I got my kernel to give equivalent quadrant scores (>2000), and it does nothing for overall usability. When running quadrant, the IO Tests are being skewed and reporting inaccurate results that give the higher quadrant scores. I don't know _exactly_ what is going on, but the difference in quadrant gives you no real world benefits other than bragging rights.


----------



## fortesquieu (Jul 20, 2011)

imnuts said:


> Just as a note, I got my kernel to give equivalent quadrant scores (>2000), and it does nothing for overall usability. When running quadrant, the IO Tests are being skewed and reporting inaccurate results that give the higher quadrant scores. I don't know _exactly_ what is going on, but the difference in quadrant gives you no real world benefits other than bragging rights.


Does your kernel fully convert system to EXT4 too? Or just data and cache?


----------



## charlie_c (Aug 18, 2011)

fortesquieu said:


> Does your kernel fully convert system to EXT4 too? Or just data and cache?


Fully ext4. Terminal shows system, data, cache, and dbdata as ext4.


----------



## imnuts (Jun 9, 2011)

fortesquieu said:


> Does your kernel fully convert system to EXT4 too? Or just data and cache?


The kernel itself will do no conversion, it just mounts whatever it finds. However, the ROM zip will either leave the partitions as-is, or format them to ext4.


----------

