Improving Headphone Testing

Measurements vs. Preferences and More!

Sound: Measurements vs. Preferences

Our proposed changes:

• Improving sound interpretation. Objective measurements are the bread and butter of what we do. However, it can be tricky to interpret this data and whether or not a pair of headphones will meet your needs. Additionally, the scale we're working on is quite large, and there's plenty to consider in ensuring our results are high-quality, trustworthy, and clear. That said, highlighting the role of preference and interpretation in certain sound tests is a good place to start.

  Tests related to the headphones' frequency response, like bass, mid, and treble accuracy, are based on (and scored against) our target curve. However, their current state may not give a good picture of whether you'll like their sound. We're not getting rid of objective measurements, though! Instead, we want to complement the data by clarifying the possibility of different interpretations depending on your preferences, which may or may not also align with our target. For example, if you like a more bass-heavy sound, our target curve framework may not make it easy to find what you're looking for.

  At the same time, preference-based tests factor into usage verdicts like 'Neutral Sound,' which are generally the first place users of all levels look to see if a product is right for them. Usages also let you quickly compare products against one another. However, some headphones can reach the same score by different means, which is a problem, especially when there are preference-based scores involved. For example, user griffinsilver216 notes that the top-scored 'Neutral Sound' headphones aren't 'tonally neutral' but are instead high performers in Passive Soundstage. This can lead to confusion over what are good-sounding 'neutral' headphones and make it difficult to trust our conclusions.

  Example	Bass Accuracy	Mid Accuracy	Treble Accuracy
  The HiFiMan Arya are audiophile headphones that match our target curve well.
  The JBL Endurance Peak 3 True Wireless are sports headphones with a bass-heavy sound profile.
  The Jabra Talk 45 Bluetooth Headset has a sound profile designed for vocal quality.

  Although these three examples are just a couple of products from our entire inventory, you can also see the bass, mid, and treble accuracy responses of all of the headphones we've tested so far, sorted by 'Neutral Sound' here.

  It's paramount to detangle our preference-based results from our scores. We can provide information on what most users like; however, it's also important to remember that sound quality is highly subjective and that there isn't a unanimous way to evaluate it. Removing scores from preference-based tests like bass accuracy can be one way to address the issue. However, understanding raw data may not be easy for everyone. We also recognize that our current resources, like graph tools and learn articles, may be inadequate for helping you make sense of the data.

  We want to implement more robust tools to make sound interpretation simpler and more accessible. We're looking into equipping the raw frequency response test with more features, like selectable target curves and normalization adjustments, based on your preferences. There's also the possibility of selecting and comparing measurements done with different test rigs. Just like with the original raw frequency test, you'll be able to compare the graph of one pair of headphones to that of any other pair in our database. In addition, we also want to shift the emphasis to bass and treble amount, which can make it easier to see if a headphones' sound aligns with your own tastes. But more on that below!

• Make it easier and more accessible to find the right sound for your needs. Since it can be hard to sift through hundreds of reviews to find the right product, we're considering adding bass/treble amount and frequency response consistency to the top of our reviews. A word scale and concrete values (based on a target curve) let you immediately see if a product is worth considering based on sound. For example, if a pair of headphones have 0 dB of bass and treble, then this indicates that the sound follows a target that research shows most people will like.

  In addition, multiple reseats of the same headphones on the same person can result in different experiences. This is where frequency response consistency (FRC) comes in. FRC shows how sound deviates based on fit, positioning, and seal. Thick hair or wearing glasses can also disrupt the headphones' seal on your head and lower bass delivery. Color-coded FRC values can thus show whether a sound profile is (or isn't) representative of all users. A red score indicates that the consistency of our measurements is low and that our sound measurements can be different from your experience. Conversely, a green score indicates a consistent experience across reseats.

  
  Example of the introduction to the Razer Kraken V3 X review.

  We don't want to dictate what 'good sound' is. However, we can make sound more accessible, simple, and transparent for users of all knowledge levels so you can see whether a pair of headphones' sound is right for you without diving deep into the data. That said, more advanced users can use these two stats to filter headphones based on the general overview of the sound profile and go further into more detailed results. This change allows you to compare multiple headphones with more control using a better graph tool.

• Further highlight the role of performance-based tests. In addition to subjective, interpretive components of sound, there are concrete aspects worth showcasing that impact your buying decision. Performance-based tests like Frequency Response Consistency, Imaging, or Peaks/Dips aren't as subjective as preference-based tests like Bass Accuracy. Instead, they're more indicative of a pair of headphones' overall sound performance, and the desired outcome is more consistently agreed upon across users. Another way to see this is that bad results in a performance-based test can be a deal-breaker for most people when buying a pair of headphones. If we reduce the emphasis on our interpretation of preference-based tests, we should also reinforce the value of our current performance-based tests on user experience.

  

  The FRC of the SteelSeries Arctis Nova 7 Wireless is sub-par. They're prone to a lot of inconsistencies in bass and treble delivery and are sensitive to fit, positioning, and seal.

  

  The FRC of the Jabra Elite 7 Active True Wireless is outstanding. They're in-ear headphones, and once you get a good fit and seal, you'll experience consistent sound delivery.

  For example, Peaks/Dips tells us how well a pair of headphones follows its own sound profile. We flatten the general trend of the sound profile into a straight line and then compensate the headphones' frequency response against it, revealing peaks and dips. A good pair of headphones have a fairly flat response, but if there are a lot of peaks and dips, the headphones struggle to control their sound profile. Most people don't like large deviations in their headphones' sound profile, so a highly variable sound is a significant drawback. Peaks/Dips is only one example of a performance-based test, but another example is Frequency Response Consistency. Although we've already covered its importance above, ensuring that users understand the impact of these tests is beneficial in trusting our results and making buying decisions.

  

  The Avantree HT5900 with poor peaks and dips.

  

  The MOONDROP KATO with excellent peaks and dips.

What do you think?

• What's your current impression of our sound testing? Where can we improve?

• Should tests that utilize our target curve (like bass, mid, and treble accuracy) be scored?

• How would you like to see a pair of headphones' sound reflected throughout a review?

• What kind of tools/features would you like to see to help improve sound interpretation?

• Are there usages you'd like to see (new usages or reworked ones)?

Test Equipment

Our HMS II.3's ear plate. It has a pinna that's larger and stiffer than a human ear.

Human ears with an in-ear microphone. Human pinnas are softer and vary in shape and size.

We've investigated several different audio fixtures, and in the end, we've purchased the Brüel & Kjær (B&K) 5128-B, though consideration was also given to the GRAS 45CA-10. Although other reviewers have used these rigs for years, we're not quite playing catch-up to them. Throughout time, we've amassed a large database of headphones tested with a standardized method, allowing you to directly compare two headphones with accurate data. That said, new rigs can help us further improve our tests, which you can then compare on a scale, unlike most other reviewers. Ultimately, we determined that the B&K head was able to provide the high-quality data needed, thanks to its more accurate ear canal and head representation.

The B&K 5128-B

The B&K 5128-B high-frequency head simulator.

The B&K’s type 4620 ear simulator is designed with a more human-like ear canal.

The GRAS 45CA-10

The GRAS 45CA headphone/hearing-protector test fixture.

The GRAS with base plate.

The GRAS with anthropometric pinnae.

Each rig offers different pros and cons, though, and there isn't one rig that's better than the other. However, a collective system can help account for the difference in philosophies and approaches between fixtures. Each rig helps us improve our tests, some of which are unique to us. Once we get them in, we look forward to sharing more about them with you!

Noise Isolation and Adaptive ANC Technology

As user thomasmusic55 also describes, these ANC systems can take time to adapt to a particular sound. Isolating single-frequency bands may not give the adaptive ANC enough time to adapt either, which can result in a worse performance. While the results are comparable between headphones, we need to do a better job of representing the product's actual performance. In response, we've tried lengthening our sweeps to see if there's a change over time. Some headphones, like the AKG N700NC M2 Wireless, perform better with a longer sweep as the ANC system has more time to adapt. However, the adjustment occurs incrementally by small frequency ranges. In real life, the ANC would be exposed to multiple frequencies at once and for longer periods. To account for this, we've tried testing a couple of headphones subjectively in dynamic environments like a subway station to check whether these impressions and measurements line up. However, these methods aren't perfect and are temporary solutions to a larger issue.

The AKG N700NC M2 Wireless' adaptive ANC's performance when using our regular sweep that lasts roughly 11 seconds.

The AKG N700NC M2 Wireless' adaptive ANC's performance when the sweep is lengthened to over 87 seconds.

Our proposed changes:

• Use real-life examples. It's worthwhile to refresh our current audio samples and examine new ways of capturing noise isolation. Sounds like plane noise or the noise of AC units, for example, are constant and predictable (in frequency, amplitude, and duration) and are more representative of day-to-day life than a sine sweep.

What do you think?

• How can we improve this test?

• Are there real-life sounds you'd like us to consider using?

What's Next?

This framework is the first step toward change. While these processes take a long time, we've already purchased new test rigs, which allow us to generate reliable as well as more current data closer to the human experience. That said, there are even more things to consider when improving our methodology—like adding LC3 codec latency or testing transparency mode. We can't do this alone, though. We want to hear from you: how can we improve our testing and methodology? Are there things we haven't covered here that you want to see in a new test bench? Please write to us below and let us know your thoughts!

Recent Updates

September 25, 2024 We've updated the article to clarify that we purchased the B&K 5128-B.

April 24, 2023 Added the 'Recent Updates' section and clarified how improving sound interpretation complements our objective measurements.