#1 - Out of Spec Policy

In the next few weeks, we will post one update every week about either a new feature on the website or an improvement in how we do things. See the full list.

The first update is our new out-of-spec policy, which aims to increase the probability that our reviews accurately represent what you can buy at home.

Our main goal at RTINGS.com is to help you find the best product for your needs. This means we want our reviews to represent what the majority of you would get. This is why we buy our own units to test. We don't want the best-case scenario, like websites that receive cherry-picked units from manufacturers, but we also don't want an exceptionally bad unit if we get unlucky with the one that we bought. Ideally, we would test a large sample size bought from a wide range of retailers, to calculate the deviation of each measurement. Unfortunately, this is not a financially realistic solution for an independent company like us.

We already formalized a policy about defective products, but we didn't have one about working products that are worse than the specifications. Here are these policies formalized:

Defective Policy

We consider a unit defective when the product isn't usable. For example, a physically broken screen, a partially broken LED backlight, or a non-responding driver on a pair of headphones. When this happens, we don’t test the unit. We return it and get a new one instead.

This policy isn’t new. We have been doing this for a few years. For example, in 2016, 2 out of the 44 TVs and 1 out of the 122 headphones we bought were defective. When this happens, we don’t publish the review. We return it and get a new one instead.

New Out-Of-Spec Policy

We are leaving the responsibility to define what out-of-specifications are to the brand, since they're aware of how their product should perform and there's no good way for us to know if our unit is worse than average.

As soon as a brand tells us the unit we tested is out of spec:

1. We update the review to mention on top that our unit might not be representative
2. We buy another unit from a different retailer to improve the chances of getting a better one
3. We retest unit #2 through all the tests affecting the Mixed Usage score and post all the results on the review page

This should remove the out-of-ordinary bad units that we could potentially review. To prevent manufacturers from calling out-of-spec on all reviews, forever, until we get a sample (best-case) unit, we apply this rule:

Conclusion

Our hope with this new policy is to improve the accuracy of our measurements and to better represent what you can buy. Hopefully, brands will start pointing out the units we have that are outliers. With this policy, you'll know that if a brand didn’t tell us that a unit is out of spec, then they don’t think it performs significantly worse overall than what you can get at home.

However, this doesn't solve the issue of us getting a better-than-normal unit, but we have a few ideas on how we could address this in the future.

If you have any feedback on this new out-of-spec policy, or maybe you have an even better solution to propose to help make sure our units are a good representation of what you can end up with, you can send us an email directly at admin@rtings.com.