Our main goal at RTINGS.com is to help you find the best product for your needs. This means we want our reviews to represent what most of you would get. This is why we buy our own units to test. We don't want the best-case scenario, like websites that receive cherry-picked units from manufacturers. Still, we also don't want an exceptionally bad unit if we get unlucky with the one we bought. Ideally, we would test a large sample size bought from a wide range of retailers, to calculate the deviation of each measurement. Unfortunately, this isn't a financially realistic solution for an independent company like ours.
Here are our formalized policies:
We consider a unit defective when the product isn't usable. For example, a physically broken screen, a partially broken LED backlight, or a non-responding driver on a pair of headphones. When this happens, we don’t test the unit. We return it and buy a new one instead.
This policy isn’t new. We have been doing this for a few years. For example, in 2016, 2 of the 44 TVs and 1 of the 122 headphones we bought were defective.
We are leaving the responsibility to define what out-of-specifications are to the brand since they know how their product should perform, and there's no good way for us to know if our unit is worse than average.
As soon as a brand tells us the unit we tested is out of spec:
This should remove the out-of-ordinary bad units that we could potentially review. To prevent manufacturers from calling out-of-spec on all reviews forever until we get a sample (best-case) unit, we apply this rule:
If Unit #2 is significantly better than Unit #1
Our hope with this policy is to improve the accuracy of our measurements and to better represent what you can buy. We encourage brands to point out the units we have that are outliers according to their expectations. With this policy, you'll know that if a brand didn’t tell us that a unit is out of spec, they don’t think it performs significantly worse overall than what you can get at home.
However, this doesn't solve the issue of us getting a better-than-normal unit, but we have a few ideas on how we could address this in the future.