Get insider access
Preferred company store

#1 - Out of Spec Policy


In the next few weeks, we will post one update every week about either a new feature on the website or an improvement in how we do things. See the full list.

The first update is our new out-of-spec policy, which aims to increase the probability that our reviews accurately represent what you can buy at home.

Our main goal at is to help you find the best product for your needs. This means we want our reviews to represent what the majority of you would get. This is why we buy our own units to test. We don't want the best-case scenario, like websites that receive cherry-picked units from manufacturers, but we also don't want an exceptionally bad unit if we get unlucky with the one that we bought. Ideally, we would test a large sample size bought from a wide range of retailers, to calculate the deviation of each measurement. Unfortunately, this is not a financially realistic solution for an independent company like us.

We already formalized a policy about defective products, but we didn't have one about working products that are worse than the specifications. Here are these policies formalized:

Defective Policy

Defective Policy

We consider a unit defective when the product isn't usable. For example, a physically broken screen, a partially broken LED backlight, or a non-responding driver on a pair of headphones. When this happens, we don’t test the unit. We return it and get a new one instead.

This policy isn’t new. We have been doing this for a few years. For example, in 2016, 2 out of the 44 TVs and 1 out of the 122 headphones we bought were defective. When this happens, we don’t publish the review. We return it and get a new one instead.

New Out-Of-Spec Policy

Defective Policy

We are leaving the responsibility to define what out-of-specifications are to the brand, since they're aware of how their product should perform and there's no good way for us to know if our unit is worse than average.

As soon as a brand tells us the unit we tested is out of spec:

  1. We update the review to mention on top that our unit might not be representative
  2. We buy another unit from a different retailer to improve the chances of getting a better one
  3. We retest unit #2 through all the tests affecting the Mixed Usage score and post all the results on the review page

This should remove the out-of-ordinary bad units that we could potentially review. To prevent manufacturers from calling out-of-spec on all reviews, forever, until we get a sample (best-case) unit, we apply this rule:

     If (Mixed Usage #2 - Mixed Usage #1) > 0.1

  • Unit #2 is better, so the review is updated with unit #2 measurements


  • Unit #2 isn't significantly better, so the brand loses its right to call out another unit until we review 10 more units from their brand

We chose ‘Mixed Usage’ since this rating is the one that encompasses the majority of our tests and is meant to represent a normal customer scenario with multiple usages.

The 0.1 threshold is to simply account for rounding errors.

The 10 review handicap was chosen since we test about 10 products per year for major brands.


Our hope with this new policy is to improve the accuracy of our measurements and to better represent what you can buy. Hopefully, brands will start pointing out the units we have that are outliers. With this policy, you'll know that if a brand didn’t tell us that a unit is out of spec, then they don’t think it performs significantly worse overall than what you can get at home.

However, this doesn't solve the issue of us getting a better-than-normal unit, but we have a few ideas on how we could address this in the future.

If you have any feedback on this new out-of-spec policy, or maybe you have an even better solution to propose to help make sure our units are a good representation of what you can end up with, you can send us an email directly at



Be part of the most informed community and take advantage of our advanced tools to find the best product for your needs.
Join our mailing list:
Become an insider

Gain early access to all tests results for new products

Early Access UI

Prices will now be listed directly in reviews and tables

product prices UI

Have 5 votes per product category to choose which product we’ll review next

Additional votes UI

Gain unlimited access to detailed test results in tools and results (no blurred results)

test table UI