Test Bench 2.1  
Changelog

 3
Updated 
A new judder photo showing the consistency of frame times depending on the signal.
Our new judder photo taken from the LG G5 OLED.

See the previous 2.0 changelog.

Goal

TV 2.1 is the first of three new test methodology updates that focus specifically on motion handling when watching low frame rate content such as movies and TV shows. Even though we've previously tested several different aspects related to motion, we really wanted to take a more in-depth look, since our previous tests didn't look at the subtleties of cinematic motion handling that can make motion look different depending on the TV. After months of research and feedback from our users, we've come up with a plan to tackle cinematic motion, and this update is the first step of that plan. We renamed one of our tests to make it more self-explanatory and to better separate it from other motion tests, added judder tests for 25 fps content, and added a completely new micro-judder test category. With these updates and new additions, we're on our way to figuring out the motion problem once and for all.

What's New In 2.1

  • Added a new Micro Judder section.
    • 24p Tests: Micro Judder-Free 24p, Micro Judder-Free 24p via 60p, Micro Judder-Free 24p via Native Apps.
    • 25p Tests: Micro Judder-Free 25p, Micro Judder-Free 25p via 60p, Micro Judder-Free 25p via Native Apps.
  • New 25p Judder tests and a scoring change.
    • Added three new 25p Judder tests: Judder 25p, Judder 25p via 60p, Judder 25p via Native Apps.
    • Scoring change: We now score the evenness of frames on TVs that don't remove judder for more nuance between TVs.
  • 'Stutter' renamed 'Response Time Stutter.'

Detailed Changelog

Micro Judder

This brand new test looks at whether or not a TV suffers from dropped frames during scenes with complex motion. Although subtle, these dropped frames are an important piece to the overall motion puzzle. Micro-judder tends to only affect busy scenes with lots of motion. Through our testing, we've determined that micro-judder is typically caused by a TV's judder removal feature. A TV can be completely free from normal judder, but can still suffer from occasional dropped frames, which cause little blips in motion that should look smooth. We have two videos we use for this test: one for 24p and a different one for 25p. Similar to our judder tests, we look at three different types of signals for each frame rate. We run each clip using a TV's native apps, via a 60p signal, and via a 24Hz or 25Hz signal. This gives us a full picture of the types of signals that suffer from micro-judder. Check out our article to learn more.

Our brand new micro judder test box.
Our new micro-judder test box.

Judder

Our Judder box now includes tests for 25p. Foreign content is popular on streaming services like Netflix, and content filmed in PAL regions like Europe is often presented in 25p, even in NTSC regions where 24p is the standard. These three new tests look at whether or not a TV can remove judder from 25p content being sent to a TV in different types of signals. The 'Judder 25p' test looks at whether or not a TV can remove judder from 25p content being sent in 25Hz, like from an Apple TV with the 'Match Frame Rate' setting enabled. The 'Judder 25p via Native Apps' test is self-explanatory and looks at a TV's ability to remove judder from 25p content when using a smart TV's native apps. Finally, the 'Judder 25p via 60p' test looks at a TV's ability to remove judder from 25p content being sent in a 60Hz signal, like older streaming devices and some game consoles. Even though these new tests were mostly designed for people watching 25p content in NTSC regions like North America, the 'Judder 25p via Native Apps' and the 'Judder 25p' tests are also applicable to PAL regions such as Europe.

In the past, if a TV wasn't judder-free from a certain signal, we would simply mark it as a "No" and assign it a score of zero. However, this didn't actually look at the full picture and lacks nuance, since the jerkiness of motion isn't always the same on every TV that can't completely remove judder from certain content. We now assign a score based on how inconsistent frame times are when a TV can't remove judder from a specific signal, and these scores are weighted into our overall judder score.

The images below show the Samsung S95F OLED on the left and the Sony BRAVIA 8 II OLED on the right. As you can see, neither TV removes judder from 25p content being sent via a 60p signal. However, if you look at our new judder picture taken from our video test, you can clearly see that the frame times on the Samsung are much more inconsistent since the squares have varying degrees of lightness, which leads to very choppy motion. The Sony model doesn't entirely remove 25p judder from 60p signals either, but frame times are a lot more consistent across the board, which means motion won't look as choppy as it does on the Samsung when watching 25p via 60p.

A new judder photo showing the consistency of frame times on the Samsung S95F and how that impacts its score.
The Samsung S95F's judder box shows very inconsistent frame times for 25p via 60p.
A new judder photo showing the consistency of frame times on the Sony BRAVIA 8 II and how that impacts its score.
The Sony BRAVIA 8 II's judder box shows more consistent frame times with 25p via 60p, despite not being able to completely remove judder from the content.

Response time stutter

We didn't change our stutter test or scores, but we renamed the box 'Response Time Stutter' for clarity. It's very common for the terms "judder" and "stutter" to be used interchangeably, but the two terms refer to different aspects of motion handling. As we continue to add more motion-related tests, we felt like it was important to make a clearer distinction here since the amount of stutter on a TV is inherently tied to a TV's response time.

The old 'stutter' test is now called 'Response Time Stutter.'
Our old 'Stutter' test is now called 'Response Time Stutter.'

What Didn't Change

We've added the micro-judder test to our 'Motion Handling' usage and adjusted the weight of some other motion-related scores. We feel like our new micro-judder test and the updates to our judder test are moving us in the right direction of figuring out how well a TV handles cinematic motion. However, we don't feel like we're quite there yet, and we still plan to release two more test bench updates that focus solely on motion handling with low frame rate content. For this reason, we've kept our 'Motion Handling' usage category in the 'Broken' state, since these changes aren't enough for us to give a confident score for overall motion handling.

The motion handling category is still in a broken state.
Our motion handling category is still in a broken state.

10 TVs Updated So Far

We are retesting popular models first. So far, the test results for the following models have been converted to the new testing methodology. However, the text might be inconsistent with the new results.

54 TVs Planned To Be Updated

We are also planning to retest the following products over the course of the next few weeks: