Updated Aug 10, 2017 at 01:11 pm

#2 - Versioned Test Benches

This is update #2 in our weekly series of new features and improvements.

This week, we're doing an overhaul of our test management system to provide stability in the scores and to remove the confusion some of you have had in the past.

Our ratings change mainly for 3 reasons:

Firmware updates
New test added or testing methodology changed
Adjustments of the weights of the ratings system

Firmware updates

For firmware updates, we're going to continue to do what we did in the past and retest the potentially affected tests when a firmware update is out. The only thing we're going to change here is that we're going to add an update on top of the review to let you know that it's been retested. In the past, it was only mentioned in the affected section of the review.

Versioned test benches

The new rating system is here to solve the problem of the last two points: when a new test is added or when adjustments are made in the weight of the rating system. In the past, we did so in a non-organized way which confused some of you since you didn’t know why the scores were different.

Now:

All tests and weights adjustments are done at the same time and are versioned. A change log will also be posted so you can see what changed. For example, here is the changelog for the TV test bench version 1.1 that was published today.
The frequency of these updates depends on how mature our test bench is for a product category. For TVs, 1-2/year. For headphones: 2-3/year.
The version of the test bench that was used for a review is listed on top of the review page, next to the ratings.
Products that are still on the market and that we still have in our lab are ported over to the new version of the test bench, to keep them current. However, the text in the reviews isn't changed during the porting process, so if a score changed as part of the new test bench, the text may no longer be accurate for the new score.
Side by Side tool and the tables still list the products that were tested with the older test benches, but the scores are converted and grayed out to show that they are approximations.