Updated Jul 22, 2016
Scoring System Overhaul
Our scoring system was flawed:
- The overall score promoted a jack of all trades, master of none. Our reviews of headphones exposed that problem. For example, the Sennheiser HD 800 S scored the best for sound quality, but got a mediocre overall score because it leaks a lot . These headphones were specifically designed to leak and not isolate, with its open back design.
- The sub scores were too limited. For example, the HDR score only included three things: color gamut, peak brightness and gradient. However, a lot of other stuff matters for HDR, like contrast or local dimming.
- Not everyone agrees with the weights and what is included. As an example, sound quality was included in the overall score for TVs even though some people don’t care about this since they have an external sound system.
The solution that we have arrived at has multiple parts:
- Get rid of the overall score. A perfect pair of headphones for every usage doesn’t exist.
- Restructure the reviews around the tests, not the usages. No more ‘Movies’ or ‘Living Room’ sections, but a ‘Picture Quality’ section instead. This removes redundancies for elements that are important in multiple usages.
- Create multiple ratings. With the test results more isolated, it is now possible to separate completely the scoring from the tests. Each usage can now be scored with only what matters for that usage, with its own weights.
With this change, we are now able to add a new feature that has been requested by a lot of you since a long time: custom ratings. You can try it out here.
You can pick and choose from any of our scored test results, and assign your own weight to the things that you care about. This will produce a table that you can sort based on your own rating. For now, that tool is pretty simple, but we will add more options in the future.
This opens up a lot of possibilities. However, the biggest drawback from this change is the increase in complexity. A lot of you are already overwhelmed by all the info present on our website, which causes analysis paralysis. This won’t help at first, but this change creates a solid foundation for what is coming in the future. In the next few months, we will make a lot of improvements to make your buying decision even easier than it was before.
Also, this places more importance on the accuracy of our measurements. Therefore, in the coming months, we will continue to improve our testing and retest previously reviewed products. We’ve recently updated a few tests like peak brightness or local dimming, and a few of other ones are already planned.
As always, let us know what you think. We wouldn’t be here without your help. We are far from our goal of a perfect review website, so any suggestion can help.
You can send an email directly to the TV reviewers at firstname.lastname@example.org, to the headphones reviewers at email@example.com.
We are also hiring. If you like where we are going, check the job posting here.