Can Consumers Tell?
Next Wednesday, I’m involved in a research project on high-resolution audio. I’ve managed to get myself more involved than I originally intended as the supplier of the source files, the audio engineer in charge of setting up and testing the equipment, and the person making the selections between the file types. The organizers hired an outfit out of the Washington DC area to conduct the research. The basic question they want answered is whether the general public can tell the difference between a standard 256 kbps MP3 file and a 96 kHz/24-bit source recording. That’s it.
I volunteered to assist by supplying a few of my high-resolution files. It’s important…in fact, essential…that the participants are presented with a real difference. Unlike the Boston Audio Society study and resulting AES paper from some years ago that used standard definition recordings vs. CD-resolution down conversions of the tracks. It should have come as no big surprise to the BAS to discover that no one could tell the difference between the “so-called” high-resolution albums supplied by members of the society and the CD versions…there was no difference in fidelity since virtually all of them were sourced from older analog tape masters! Knowledgeable audio people still cite this highly flawed study as “proof” that high-resolution audio is irrelevant. I saw a post just this past week.
Anyway, this new research project will at least be starting with files that provide very different levels of fidelity to the listeners. I took the very same files that Scott Wilkinson used for his non-rigorous study through AVS Forum last summer and I converted/compressed the 44.1 kHz/16-bit CD spec files to 256 kbps MP3 format. As you may recall, those members of the AVS community that had systems capable of high-resolution audio playback did beat the odds and were able to identify which file was which. The number of individuals that participated was very low and no real conclusions could or should be drawn from the casual study, but it was an interesting effort.
I don’t expect much from the new study. It will be conducted in a Beverly Hills research facility, which is basically a business type conference room with glass walls (including a one-way mirror for observation) and a large table in the middle of the space. Teac and Sony are providing the playback equipment. I have the amplifier, headphone amp, and DAC from Teac. The speakers (retail for around $250 each) claim to be able to deliver better than 20 kHz. We’ll also be using headphones…although I don’t know the model at this point.
The files are ready. I’ll post them to the FTP site so that all of you can challenge yourself. They are all 96 kHz/24-bit PCM stereo files (so that they can be played simultaneously and be randomly selected) but the quality of audio ranges from the original source, to CD spec, and finally 256 kbps MP3. The levels are matched. I will be using my laptop running Pro Tools 10 and switching between the stereo tracks using the SOLO buttons on the virtual console. The switching is completely seamless and noise free.
When I listen to these files sitting at my computer, I can’t tell the difference between them. In the studio, I can…without fail (I got 5 out of 5 on the ABX test at AVS last summer). Will the people that are participating in the study be able to tell them apart? My guess is that they won’t. The results of the study will find that the general public is unable to perceive any difference between a good quality MP3 file and a bona fide high-resolution original. Then where do we go?
You can visit the FTP site and check out the files for yourself. I will randomly label them A, B, and C. I’d love to hear from readers about what you hear. Please don’t cheat and look at the spectra etc. Just listen to them and send me an email identifying which one you believe to be the source, which is the CD version, and which is the MP3 file. I’ll collect the information and report back. It would be great to know what system you used to audition the files.
+++++++++++++++++++
I’m still looking to raise the $3700 needed to fund a booth at the 2015 International CES. I’ve received some very generous contributions but still need to raise additional funds (I’ve received about $2200 so far). Please consider contributing any amount. I write these posts everyday in the hopes that readers will benefit from my network, knowledge and experience. I hope you consider them worth a few dollars. You can get additional information at my post of December 2, 2014. Thanks.
Why is this test being crippled by using a set of $250 speakers? The only way this test can provide meaningful results is if the headphones used are at least at a quality capable of HD and top end responce to around 40k.
I don’t get why the test seems set up to fail?
They say that they want to test the real world case of the average Joe.
Ah, I see. But then we get back to the cost factors. As things stand now how many “average joes” are willing to pay the premium $ for HD media. ( $44.99 for JATP) LOL
Like we’re stuck in a chicken-egg thing.
I’ll give your files a test on my Sennheiser Momentums off my computers Realtek ALC889a chip. Best I can offer, My Pioneer 5.1 receiver and Klispch HT600 5.1 speaker system just aren’t up to critical listening.
Before thinking about the cost of High Resolution audio files…it would be nice to actually discover whether people can tell them from standard or reduced resolution audio. And that we have an accurate definition of what constitutes a real HD-Audio file.
I’m looking forward to testing my ears and my cheap HRA system (no component is over $500). Thanks for taking the time to share this.
I’m glad you are involved in the test. At least the sources will be known and the test fair.
As you, I love to listen to as good a music files as I can get. But, if I’m any comparison to the group that will be tested, many won’t be able to tell the difference. My 71 year old ears were punctured several times in my youth due to several bouts of strep throat and other sicknesses.
I have downloaded several files from many web sites and done my own comparisons. I even bought the AudioEngine D3 DAC and a reasonably good set of headphones. That’s as good of sound system as I’ll ever have but I can’t even tell the difference between a 128K and 320k mp3 file. All the 24bit/92khz flac files I’ve downloaded sound great, but I really can’t tell them apart from good MP3 files. So I’ve kind of decided that I won’t be spending a lot on premium prices Hi-Rez music files.
PS. I just contributed to your CES fund. Have a great time. Wish I could be there. I attended many of the summer ones when they had them here in Chicago.
Kit thanks for the comments and the contribution.
I have a system at my studio where I have sat people down and had them do a mp3 256 VBR mp3 file and a 24 bit 96k file of something like “Dark side of the moon” or “Goodbye Yellow Brick Road” which were both remixed and remastered, and everyone can tell.
Now, I have a set of Genelec 1029 monitors and a Tannoy subwoofer. This makes it easier to hear, since it has a nice image and things like reverb tails and the depth of tone of the high resolution audio becomes pretty clear.
I agree with one of the other posters: if you give them a crappy system to use as a reference, they might not hear anything due to the quality of the playback, which in my opinion should offer the audio being tested in a high quality but also transparent audio system… it’s the only way.
George Leger III
Thanks George…I find it interesting the both of the records you listed are standard resolution analog recorded masters. The move to audiophile content takes the difference even further…that’s why I’m involved in the current evaluation.
Science requires that the constants (files, electronics and transducers) be transparent, so the object can actually reach the variable being tested (the average listener). If these people do not conform, they are not serious. Being there Mark, you could gather the flaws and report on it later. One of these days, someone will do a proper comparison.
PS. I cannot do your test because my desktop audio equipment does not reproduce HRA. I only paid around 2000$.
I’m still hoping to set up a proper study at my facility sometime soon. This study is expensive…and will prove nothing.
PS. I cannot do your test because my desktop audio equipment does not reproduce HRA. I only paid around 2000$
Are you sure about that Ed, even the least expensive computers built in the last 5 years or so have built in audio chips with codecs that will do 192/24 out of the box. Download one of Marks 96/24 files and test playing it.
Good luck.
Thank you Sal.
My computer can indeed read the 192/24 files but I would need a far better audio system in my office to make the difference between a 320 kbps MP3 and even CD quality. I have made this interesting test in the past: https://www.goldenears.philips.com/en/introduction.html.
However, in my home-cinema/listening room, I clearly hear the difference between a standard 256 kbps MP3 file and a 96 kHz/24-bit source recording (from Mark) and everything in between. My point is: the audio system must be good enough to pass through the HRA (ultrasonics, quick dynamics and other goodies).
Mark, as I recall, there was a commentator named Jim who effectively disproved your 5/5 ABX success.
5 repetitions and 0% result does really mean zero right answers identifying X as A or B. 1/5 – 20%, 2/5 – 40%… 5/5 – 100%. Also, I’m not aware of any headphones that are reasonably extended in FR above 20 kHz. Oppo PM1 is not an exception – http://www.innerfidelity.com/images/OppoPM1.pdf
Based on the assumption that the difference between CD/HR is in frequencies above 20 kHz it is in my opinion highly unlikely to genuinely hear a difference using headphones. Or with speakers, for that matter.
This site usually has a very nice clear scientific narrative. It needs the same treatment concerning the supposed audible benefits of HD as an end format.
You’re right…although I disagree with his assessment. I played all of the tests and got exactly the opposite of what was correct each time…which in my mind says that I did manage to tell them apart. There’s a lot more to High-Resolution audio than just ultrasonics…the things I listen for are also tied to the extreme dynamic range of the music, which is vastly superior in a high-resolution track
If you can genuinely reproduce both 16 and 24 bit dynamic range I’d agree, but how many systems can genuinely reproduce a S/N of 144db or even 96db? BTW I’m still awaiting an approval in yesterday’s post.
Obviously, we strive for the best system we can. And there are recordings and systems that reach above CD spec sound. Not many…but some. BTW I don’t see another post here to approve. Write it again, can I will.
Re-posted:
December 14, 2014 at 8:55 am
The Lavry paper’s a good read – I’d already posted a link on this site to the his first paper (and reproduced his work in Mathcad) but hadn’t seen this paper; thanks for the link.
Sony has a full line of phones on the market now that are claimed to be HD ready, including the very high end model with a stated response of 3 – 100,000 HZ.
Can’t speak to the accuracy of their claims but the HD market has drawn the attention of speaker and headphone manufacturers and products are coming to market that are designed to have HD capability.
I’m hoping that the Sony folks will provide some “HD” phones.
Dear Mark,
Thank you for sharing these test files. I am going to do my own test this week and sending back the results.
Based on my personal experience, sometimes I (and some of my friends with appreciated ears) could not make difference between MP3 and CD quality – but MP3 was 320kbps. Depending on the type of the music (e.g. in case of “simple music”) sometimes it is either hard to distinguish the CD from 24/96. Even on a good audio chain. But in case of “complex enough” music it was always obvious which is what. The 250kbps MP3 vs 24/96 shall not be a question with a good playing system.
The key is also in your post “In the studio, I can…without fail”. Only a proper and matching playing system will benefit from a better source. An “average” car will not drive better even if you fill it with super fuel…
What is their definition of the “real world case of the average”?
Me too did not understand the $250 speaker (you answered already), but I also do not understand the 250kbps resolution. MP3 is capable of 320kbps, either iTunes downloads can have this rate.
Why the resolution is constrained to 250kbps?
What is the aim of the experiment/test: do they compare the formats (MP3 vs whatever) or the low resolution vs high resolution? In case of format why they do not choose closer resolutions, in case of resolution why they do not choose the same format?
What is comparable at all in this test?
As you wrote: “I don’t expect much from the new study.” Neither me.
I do my own test however with certain expectation, as this is the first time for me, when I can compare such a different sources when I can be really sure, they came from the same record and sampled/converted with the same method/equipment/ears.
Thank you for the opportunity!
best regards
Andras
I think the aim of the test is to spend the allocated funds on a research project. They clearly don’t have the knowledge nor have they hired the right team to pull off a real test.
Mark wrote: “I think the aim of the test is to spend the allocated funds on a research project”
Given the nature of blue sky research, we both know what that will achieve..