HD-Audio Challenge II: Preliminary Results
The HD-Audio Challenge is coming to an end. Tomorrow is May 1 and it’s time to wrap phase one of this research project and begin the process of analyzing the raw data. It will be a long and arduous task — one that I have to admit I’m not equipped to do on my own. Thankfully, I have heard from a number of statisticians who have offered advice and help. One very kind and experienced gentleman has already spent time preparing a custom spreadsheet that provides the ability to parse the data as needed, get answers to specific questions (Does the level or cost of equipment affect the outcome? Is age a factor?), and graphs the data in a clear manner. I can’t say that I’ve gotten comfortable with his document but I will be digging deeper as I continue the project.
For anyone still wishing to participate, there’s still time to request the files, do some critical listening, and submit your results. Simply visit the post with the sign up form: https://www.realhd-audio.com/?p=6713 and I’ll send you the link to the files.
To date, I’ve had almost 2000 music lovers submit a request to participate, over 1000 people have confirmed receipt of the 40 files, and 360 submit their results. I would like to encourage everyone that has spent time with the files and tallied their results to visit the submission page and send me their selections. The more people submit their results, the more valid the data. Here’s the link to the submission page: http://www.realhd-audio.com/?page_id=6746.
I’ve also had a number of friendly email exchanges with other interested individuals offering their thoughts about the survey. When I wrote that the average score was 40% in a previous post, a gentlemen wrote:
“If people average 40% correct, then they can actually hear a difference with some statistical significance but picked the wrong one as the high-res. If no one can hear a difference, then while individual variances can be large (even if they just tossed a coin to select), over a large enough population, the average should converge to 50% not significantly lower, not significantly higher. Otherwise, your thesis will not be supported.”
So I went back to the raw data and did a little additional calculation because Robert, the individual behind the graph below, had indicated that the number was closer to 50%. First, I removed anyone that could not confirm they actually listened at 96/24 or knew that their system downconverts to 48 kHz. I also removed anyone that used any type of analytical tools or other non-listening methods to make their selections. Finally, all of the people that responded that didn’t perceive any differences between the native 96/24 master and the downconverted 44.1/16-bit Redbook standard files, were counted as 50% (there were 44 submissions that stated “No Choice” for all tracks).
The average success rate — across 350 people — in selecting HD-Audio vs. CD-Audio was 47.7%. That means that people did no better than a coin toss. And it didn’t make any difference if they had a “highly resolving” system costing over $50 or simply listened on headphones from their portable DAP. There’s obviously a lot more analysis to be done but the initial take away from this survey shows that HD-Audio or “hi-res audio” as promoted by the CTA/CEA, music industry, equipment manufacturers, and magazine publishers is not worth the investment.
The gentleman that initially pointed out that 40% might actually be statistically significant added that my interpretation of the data might be subject to “confirmation bias.” He stated that I have a vested interest in the outcome and therefore should steer clear of making claims prior to a full analysis of the data. I responded that, in fact, by seemingly refuting the value proposition of hi-res audio, I diminish the value of my own catalog of native 96/24 recordings. For the longest time, I wanted to believe that HD-Audio made a difference. I was among the strongest advocates for this new and exciting “upgrade” to audio reproduction. When Paul McGowan states emphatically that “Oh God yeah” high-resolution audio is easy to identify, he’s just plain wrong…and so are many others.
After the first HD-Audio Challenge seem to indicate that people even trained listeners with above average systems — couldn’t pick out an HD file from a Redbook CD, I began to have serious doubts about my previously held position. The hundreds of people that have participated in the second round of the HD-Audio Survey, have confirmed the results of the previous project. It is no longer possible to claim that “hi-res audio” is an important next step in the evolution of audio. HD-Audio is completely unnecessary for the reproduction of hi-fidelity. It is a very good thing to record using 96/24 PCM audio but for the distribution of music, it’s nothing more than a sales slogan.
Over the next few days and weeks, I will be revealing more about the survey, going through all of the tracks, and discussing how we move forward from here. But when someone posts on a FB group or in a YouTube video that it’s easy to tell a hi-res audio tracks or album from a standard-res version, ask the person if they participated in the HD-Audio Challenge II. If they haven’t, you might gently suggest that they download the files and see if they can tell them apart.
John Gorka Broadcasts New Tunes…From Home.
One of my favorite artists is John Gorka. A few days ago, I received an email from his manager with a link to a new COVID-19 themed song: https://www.facebook.com/watch/?v=260584415139849 recorded and performed by John. It’s called, Soap Opera and tells the now familiar story of a musician’s plight in a time of “stay at home” mandates.
The next day the manager wrote again and bragged about another video of John playing a live concert in Santa Ynez back in 2013. The performance is available on YouTube and DVD and shows John doing what John does best — perform his songs with heart and soul. I guess I was tweaked because David, the manager, was wrote that the video is, “the best live footage of John..ever. Great video, great audio….and, of course, great performance.” I beg to differ. The “live” session we captured of John about 10 years ago is IMHO far better, has audiophile quality audio (in stereo and surround), and isn’t compromised with audience noise and applause.
The Gypsy Life is available as a Collector’s Edition double DVD set or on Blu-ray. The deluxe version has interviews, behind the scenes footage, a discography, and a live concert from McCabes here in Southern California.
In fact, I heard from my friend Hale Milgrim recently. Hale produced a music program called “Sings Like Hale” for a radio station in Santa Barbara and was the head of Capitol Records years ago (among other executive positions in the music industry). He’s one of the nicest men I know and one of the last of the real “music” men (as opposed to MBA types) to run a record company. He called to say hi, check up on things, and thank me again for the audio/video productions that I’ve produced on AIX Records. He cherishes them and is particularly fond of the one we did with John Gorka. Praise from Hale, someone that has seen and heard it all, is a high honor. And the reviews that his disc has garnered are over the moon.
Here’s a couple of comments from customers:
“You had ask me to let you know what I thought of the John Gorka disk after I had received it and listened to it. I must tell you that I was more than blown away by not only the undeniable sonic quality of the disk itself but by John’s endearing and heartfelt lyrics. I am now a firm believer in AIX records because with the pure passion and cutting edge technology you bring to the market. I only wish other studios out there had your desire to produce “quality” music as I can only imagine what sonic pleasantries we all would have.“
“I have never purchased an AIX product before but after reading Ralph’s review of the John Gorka disc I decided that I would give it a try. I paid about 13.00 postage to get the package to Australia. I am blown away by the clarity of this recording.”
The 30% discount is still in effect for all AIX Records purchases. Use the coupon AIX30Percent during check out.
Hi Mark,
Whilst this is an early comment following your pre-result blog post, I am dismayed to read that 47% is considered no better than the toss of a coin in terms of a statisical result.
Far be it from me to debate the comments from the statistician, it seems to be a lot of work to be dismissed as not having any validity because the result is around the 50% mark.
For those that believe they can tell the difference, but be told their input basically holds no meaning is a bit harsh.
I decided not to take the test on the grounds that there simply isn’t enough provenance available to tell me whether recordings are made at a high rate in the first place and that the majority of the music I listen to was made long before recording at high levels became a thing.
I have no doubt in a blind test I would not be able to tell the difference, simply because I have a remastered CD of several albums and a Hi Res version of the same and cannot tell a difference between them.
Following some of your previous blogs stating that the vast majority of music available never made it through a Hi Res production process, it has become mute as to whether a recording I buy is in a 24 bit container, I only buy them if I know that the remastering process has yielded good results from visiting various forums, or I happen to know that the original sounded good in the first place.
From my perspective your survey is only going to prove to the recording indusry that the production process is not worth upgrading and that High quality Hi Res recordings will remain the preserve of the likes of 2L.
I look forward to the detailed analysis of your work and the subsequent dialogue. It is such a shame that Hi Res will only be a benefit if it is used for the recording process and that I know the majority of the music I like will not use that process.
But when it comes down to it, I will continue to enjoy music and not fret over the smapling rates or word depths.
And of course I will continue to seek out great recordings likewise undisturbed by their resolution.
Hope you are keeping well at this time and look foward to your posts on this subject.
Mark,
I just signed up for the survey and received access to the files. I am starting the listening…
Upon reflecting upon the methodology of the survey, I am questioning (respectfully) its validity?
What the survey is asking us to do is to compare is a 96/24 master Hi-Res audio file to the same, however downconverted to a 44.1/16 CD Redbook format and then upconverted back to a new 96/24 PCM file (for sake of a better blind test).
The result is that the downconverted / unconverted file is a copy exact of the master Hi-Res up to 22.050 MHz.
The portion of te spectrum from 0 to 22.050 MHz is the same in the two files we are asked to compare. Hence why it seems that the participants have a hard time discerning between the two.
So it seems to me, but I may be wrong, that the survey is more about who can hear sound above 22.050 Mhz (which most of us human past the age of 25 cannot), as opposed to whether a High Res audio recording is of better audio quality than a Redbook CD audio recording.
I wonder if a better survey would be to compare a true Redbook CD recording to a 96/24 Hi-Res audio recording? I understand that the file size difference may make a true bling comparison difficult, however, I believe this is what the survey is trying to get to.
What am I missing?
Philippe, you forgot that the two primary differences between a high-resolution file and a Red Book track are the sampling rate AND the word length. So a native high-resolution audio recording converted to a CD Red Book standard-resolution file is about the fidelity differences between them…if any. Putting the CD spec fidelity back in a 96 kHz/24-bit PCM bit bucket doesn’t change the fidelity any more than digitizing a vinyl LP at 192 kHz/24-bits helps the fidelity.