The Results Are In
As many of you are aware, my friend Scott Wilkinson (the Home Theater Geek and editor in charge of AVS Forum) and I concocted a casual test to see if users equipped with high-resolution audio capable playback systems could perceive a difference between a standard resolution, CD spec audio selection and the exact same track in real high-resolution (96 kHz/24-bits). It was never intended to be a definitive or scientifically rigorous comparison but we felt it was worth doing. One of the reasons is because many HRA doubters point to the Meyer and Moran study that was written up in the AES Journal as proof that human listeners can’t distinguish between a 44.1 kHz/16-bit down conversion of a “high-resolution source” (they are not alone in reaching this conclusion).
It surprises me that all of the studies that have done these comparisons start with source recordings that don’t contain any fidelity above that of a compact disc. I’ve written about this previously in a post of two. So Scott asked whether I would be interested in using a few of my tracks (which actually do contain better than CD fidelity) in a non-rigorous test on the AVS Forum. It took a little work to figure how we would prepare the files and present them to the AVS readers, but we figured out a scheme and posted the files. Some of you may have gotten involved and submitted your results.
Scott collected all of the submissions, did some statistical analysis, prepared some graphs, and came to some preliminary conclusions. And things seem to confirm that when listeners with high-quality playback systems listened to real HD-Audio tracks, they were able to identify them 85% of the time. Here’s a quick summary of the results. You can tune into the Home Theater Geek show later today (9/11/2014) and hear an hour-long discussion about our little study.
The first thing that Scott did was divide listener’s system into two groups. There were those that were considered HRA capable and those that weren’t. People were asked to describe their equipment and provide specs if possible.
Here’s a graph showing the success rate for HRA and Non-HRA capable systems.
Figure 1 – This is a graph showing the number of correct responses for HRA systems and non-HRA systems. [Click to enlarge]
The red bars are the non-HRA responses. Notice that the x-axis ranges from “No determination”, meaning the listeners couldn’t tell any difference, to three, which was the number of tracks in the test. The blue bars are those with HRA capable systems. These folks did much better. The non-HRA systems results were around 50%, indicating no ability to tell the difference.
However, those with good systems (and seemingly good ears) faired much better. They average about 85% correct in the test…statistically significant.
Here’s another chart showing the success rates for each of the individual tracks.
Figure 2 – Accuracy by track. [Click to enlarge]
This test involved a very limited number of participants and was done by individuals in their own homes. We make no claim that the results are meaningful or answer the question. But it is encouraging that with REAL HD-Audio content, we got results that agreed with the fidelity of the source files.
I’m seriously looking to do a rigorous test using JBL M2 Studio Reference Model speakers in my studio with my state-of-the-art equipment. I want 500 participants from all sorts of backgrounds. We’ll see what happens then.
Hello Mark, I’ve been subscribing for about a month, and am interested in most of what you write. I’m especially interested (my background is R&D engineering) in the previous tests. If I read your description correctly they compared regular systems to 24/96 systems with redbook CD material only? Holy cow. No wonder you are so up in arms. What exactly do they think they are proving? Oh well, people have been lying with “research” and “statistics” since before you and I were born, and they will be doing it long after we are gone. Thanks for shining the light where it needs to be focused.
Now, I just need to read up on your take on DSD and get my head around that…..
And thanks for the explanation of the difference between 16 and 24 bits. After I read it a couple of more times I may understand it. But so far it makes sense (simplistically, aren’t we really just dropping the background noise? -excuse if that’s not exactly the right word).
All the best – John P
Check out the post on the BAS study here. Essentially, all previous tests are using content that the labels claim is ‘high resolution”…like SACDs and DVD-Audio titles. In fact, a transfer of an older analog tape with limited frequency response and low dynamic range has less fidelity than a CD, in most cases. The SACDs that were used as sources came from analog tape….so the comparison was impossible. A downconverted analog transfer to 1-bit 2.8224 MHz SACD isn’t going to have more fidelity thatn a PCM CD.
You’ll find my assessment of DSD to be decidedly cool…to put it mildly.
You got the essence of the benefit of moving to 24-bits.
Mark,
these results certainly look encouraging. But as you wrote yourself, the number of respondends was quite low (33 non-HRA and 23 HRA systems if I read the diagrams right. And the systems were all different.
This got me thinking: you are teaching at uni. Did you think about starting a scientific study there? You could set up two systems and ask your students to volunteer as test listeners… 😉 Or suggest this as topic for a thesis paper?
Just an idea…
Best regards,
Oliver
I’m creeping closer to making this happen. In fact, I was in touch today with the guy at JBL about get their M2 Studio Reference Monitors in the studio…they can reproduce 40 kHz. We’ll make this happen.
If I am in LA when you do it, I would love to participate.
Would love to have you.
I believe a HDA recording of a good HDA master such as one done in your studio and on your equipment would be easily identifiable against a Red Book downsample, but I have this question.
Do you believe a HDA recording “say 96/24”, of any “classic rock” analog master from the 60s – 90s can offer any improvement over a well done Red Book CD?
I have a number of HD downloads from HDTracks,and their sister CD’s that I’ve ripped to flacs,and play back the digital file streams over my over the same DAC. I have yet to hear any major improvement from the HD file, nothing I think I could ID in a double blind test. I’ll put my rips of a lot of my 20+ year old MFSL CD’s against most of the stuff I’ve got off HDTracks any day.
I would never say that it’s easy…but I do believe that it is doable. The question of transferring a “classic rock” analog master to 96 kHz/24-bit PCM or higher can only offer a very slight improvement if any…it depends on the tape, the alignment of the machine, the skill of the operator etc. I’m not surprised that you don’t hear a tremendous difference between an HD track and your ripped CDs. There’s isn’t much.
Awesome – I got involved in the tests (HRA compliant system, result was 2/3) …Looking forward to tuning in 😉
Hi Mark,
I like your idea of setting up a further HRA test using the JBL M2 Studio Reference speakers. How about doing it during the fall AES Convention next month? I believe it’s not that far from your studios. I will be attending the AES Convention. I would love to participate. Please contact me at my e-mail address.
Thanks,
Rick
I’ve reached out and asked if this is interesting…will keep you posted.
I hope you do the robust test. When you do that, I hope you create a better version of 16 bit, 44.1k files than in the AVS test. The timing misalignment issue alone on certain songs made it easy to identify whether A or B was X in an ABX test and has nothing to do with quality or preference. The choice and use of the downsampler application and its settings will be crucial for credibility.
I did the SRC conversion and can assure it was done with care and the best process available. I’m not sure what you’re referring to in the timing misalignment. There is not way that the two versions will line up perfectly when they’re at different sample rates. I’ll have to go back and look at the files to see how far they are different.
With respect, either the tool you used was hardly the best, or setting were bad, or some combination. Use a free tool like Audacity (latest version) to compare the different song versions to see the timing differences (e.g., Mosaic has a 10 ms difference readily audible in ABX testing (about 4ms difference is thought to be the audible limit)) and level issues (the 0.2 dB hack applied to the files to correct for the resampling level error is readily audible in the quietest sections of songs, making the noise level discernible). Use the “effect>invert” command in Audacity to create the difference between two files and then look at the frequency response of the differences.
Then use Audacity’s built-in resampler with noise-shaped dither and repeat the resampling steps and compare. As several others have already shown, the level and timing issues are solved and overall differences essentially disappear in the frequency range common to both files for a song. The latest Audacity uses SoX, which is widely recognized as among the best resamplers.
If you don’t use a better resampling approach (noting that some of the errors will be mitigated because you won’t need to resample back up to 24/96k), I fear the new test could be readily disregarded as merely a test of the audibility of resampling errors. If you want to publish your results, and given the resampling errors widely noted in the informal test, I recommend you address this explicitly with difference graphs in any published paper to show the informal test issues were overcome.
Stuart, the Process Application and the Sonic Studio DAW are the tools of choice by professional mastering engineers. There are no others that come close with the exception of SADie and I’m not even sure they are still around.
I have Audacity and will try some of the things you mention in your comment. However, I don’t attribute the .2 dB change that we applied as the reason for the background noise difference…that’s what happens when you reduce the number of bits and the sample rate.
I’ll spend some time with Audacity and experiment with it. I recognize that it is critically important to have the best conversion as possible. The test that I will be doing here at the studio will be playing the 44.1/16 CD spec directly vs. the high-resolution file…the conversion will only be one way.
Mark,
I’ll be following this with much interest! I am waiting for delivery of a pair of JBL M2’s.
As you were, I was so impressed after listening to them I sold my old gear and bought them. They leave an impression that is hard to ignore. I’ve never heard a speaker with such clarity and resolution and I’m confident that it’s the right tool for the job.
I’m hoping to get my hands on a surround setup…we’ll see. These speakers are definitely among the finest in the world…if you want clean, accurate, uncolored sound. Congrats!
I just got back from traveling and am just getting caught up on your posts. Unfortunately, I was not able to participate in your test. Very interesting results!
Regarding the question about whether it is the reproduction of ultrasonics or something else that makes the difference, may I suggest that a comparison be conducted between 24 bit 96 kHz files in a different manner. One would have the ultrasonics removed directly by a filter in an audio editing program rather than converting to 44.1 kHz and then back to 96 kHz. Use a gradual slope filter to avoid the “brick-wall” filter sound.
Also, let me provide a little different perspective on the difference I hear between 16/44.1 and hi-res. I listened to many years of analog before CDs were introduced. The difference I immediately heard was that CDs added something very much unwanted to the sound. It’s an annoying hardness I call “CD edge”. As CDs and players have gotten better, that annoying edge has been reduced, but I still hear enough of it (even with the OPPO 105’s stereo outputs) that extended listening to CDs is fatiguing. I do not hear this edge in hi-res, even if has been made from an analog tape. This “easy to listen to” quality also applies to SACDs even though they may not contain much in the way of ultrasonics. In my option, this is a difference that may separate low-res from hi-res in addition the ultrasonics and bit depth you are testing. Perhaps it is one reason why there are differing opinions on what constitutes low-res vs. hi-res.
Roger Jones
Interesting…the much discussed CD “harshness” may have existed in the earliest CD when we had to use pre emphasis during mastering but it’s been a long time since that was a problem. Although, I do agree that there is a different feel to high resolution…real HD-Audio can be more accurate and provide dynamics and frequencies that standard def CDs cannot.