Perceptual Evaluation of HRA Study Proves Nothing!
There is small but noticeable buzz circulating among audiophile sites, FB pages of prominent audio experts, and even mainstream websites stating without reservation that high-resolution audio has been proved to be perceptible. I’ve read no fewer than 6 headlines pronouncing things like, “Hi-Res Passes Key Test” (CEOUTLOOK.COM), “It’s Official: People Can Hear High-Res” (Audiostream), and “HD audio is perceptibley [sic] better than CDs”. Robert Stuart, the man behind MQA and a staunch supporter of high-resolution audio posted “This is a landmark day for audio”, on his FB page yesterday.
These posts and articles spring from a peer reviewed AES (Audio Engineering Society) paper by Dr. Josh Reiss of Queen Mary University of London titled, “A Meta-Analysis of High Resolution Audio Perceptual Evaluation”. He wrote to a number of parties — including me — yesterday with the article attached. Dr. Reiss didn’t do a study to arrive at his conclusion that his study found “high resolution audio has a small but important advantage in its quality of reproduction over standard audio content”. He did a meta study of previous high-resolution audio studies. Here’s the abstract from his paper.
“There is considerable debate over the benefits of recording and rendering high resolution audio, i.e., systems and formats that are capable of rendering beyond CD quality audio. We undertook a systematic review and meta-analysis to assess the ability of test subjects to perceive a difference between high resolution and standard, 16 bit, 44.1 or 48 kHz audio. All 18 published experiments for which sufficient data could be obtained were included, providing a meta-analysis involving over 400 participants in over 12,500 trials. Results showed a small but statistically significant ability of test subjects to discriminate high resolution content, and this effect increased dramatically when test subjects received extensive training. This result was verified by a sensitivity analysis exploring different choices for the chosen studies and different analysis approaches. Potential biases in studies, effect of test methodology, experimental design, and choice of stimuli were also investigated. The overall conclusion is that the perceived fidelity of an audio recording and playback chain can be affected by operating beyond conventional levels.”
As a strong advocate for high-resolution audio/music, I’m encouraged by his meta analysis of the existing literature (at least the 18 studies that he included in the paper…including the infamous Meyer and Moran AES paper). But as we all know, the devil is in the details. I found the paper lacking for a number of reasons…starting with the definition of high-resolution audio.
The opening sentence gets straight to the most important issue associated with high-resolution audio/music. The paper states, “High resolution audio may be loosely defined as those systems and formats that are capable of rendering beyond standard quality audio, i.e., more than 16 bits, and/or more than 44.1 or 48 kHz sample rate, as used in Compact Disc (CD) or ‘ordinary’ Digital Video Disc (DVD) quality audio.”
Reading carefully, Dr. Reiss establishes “standard quality audio” as PCM digital audio with specifications of more than 48/44.1 kHz sampling rates and 16-bits. Already, we have a disconnect. The CTA/CEA Audio Board, major record labels, DEG, and NARAS organizations don’t endorse this definition. I fought long and hard to get the CEA board to move their definition above CD specifications as the baseline for high-resolution audio. The result was I got kicked off the board. Dr. Reiss’ definition does nothing to help solidify the “looseness”. I would suspect that a paper of this magnitude would have to have a rigorous definition in order to report any valid results. Did the studies that he examined use 48 kHz/20-bit PCM audio or something higher? 48/20 is high-resolution audio in the minds of the previously mentioned organizations.
One of my biggest complaints is the focus on the “rendering” or delivery side of the audio production chain. This completely ignores the “provenance” of the recording itself. This is the trap that Meyer and Moran fell into and is what makes their AES paper so utterly useless (but it has been used by those opposing HRA for almost ten years!). The content — the recordings used in any of the studies — has to actually contain ultrasonic frequencies and possess dynamic range higher than a standard CD. Although the paper specifically states that it didn’t analyze the papers for perceptible differences associated with greater bit depths, I believe the move to greater dynamics is more likely to be perceptible than ultrasonic frequencies.
I am familiar with many of the 18 studies that Dr. Reiss analyzed and know that with the exception of the Meyer and Moran paper, most of the recordings used were not commercial high-resolution audio recordings. Some of the best studies made their own recordings using high-resolution equipment and took great care to ensure that ultrasonic frequencies were present. It was necessary to make new recordings because there aren’t very many commercially available.
The paper acknowledges that “jazz and classical” music was generally used…not rock, folk, pop, or country music. Dr. Reiss said, “The studies that most showed an effect mainly used jazz and classical music, but this wasn’t exclusive”. So we have another disconnect. The headlines are cheering in support of the paper and “proof” of high-resolution audio’s perceptibility for a very narrow category of music while the industry (both hardware and software) continues to promote and sell recordings that no one would be able to distinguish from their standard-resolution CD versions and a high-resolution transfer into 96 kHz/24-bits.
Sometimes it’s the stuff that isn’t featured that is the stuff that matters most. The provenance of the “chosen stimuli” (the recordings that were actually played for the subjects) is the most important thing…and it doesn’t get much play in the AES paper.
I applaud Dr. Reiss for his important work. He rightly describes the kinds of things that need to done during another study into the merits of high-resolution audio. I would love to be involved in a rigorous study, but the media and many supporters have already proclaimed the myth of high-res audio, as it currently exists, as fact.
You can download the paper from the AES Journal site: Journal of the AES.
Garbage in, garbage out. Each of the studies in the meta-analysis were garbage in that the provenance of the recordings did not show at least 24/96. So what does the whole study prove – nothing. Then there’s the use of the word “perceptual”. That means subjective. How can you prove subjective without analyzing the subjects (listeners)? This is pseudo-science in the hopes of selling a group of hardware & software.
That’s my biggest issue with all of these studies. Although, the author of this paper did recognize the need for the right stimuli to be measurable at the listening position.
A falsehood spread by a majority is always more accepted than the truth spread by but a few. In the former the numbers add up to nothing but glitter. The latter is found only by those cynical enough to question the accepted.
Touche’ !
pj
This is getting to be almost as two-sided a topic as politics; each party makes what would appear to be a supportable case. We end up knowing less, not more., because “how do we know what is true anymore?”
We have previously, virtually flogged the issue of ‘provenance to death. It all depends what group you are trying to please.For the purists,(and I am one at times,) the considerations of provenance and musical frequency content do fairly apply.
For the larger group who love music and good sound but are not audio-political, simply hearing ‘first gen sound’ is as gratifying as is desired. As was pointed out on a very good video supplied to me by the good folks at Mo-Fi , that original Edison cylinder you said that I would prefer does in fact have considerably more dynamic range than most modern recordings. http://productionadvice.co.uk/research-loudness-sales/ . So ironically, I would have a sound reason to prefer that ‘first gen’ recording. Best,Craig
” May be loosely defined”….. My, sounds like politics to me ! Control the language, control the “crowd”. Social media is the wateland of pseudo science. Credence is hard to find anywhere these days. Good luck with this one Mark,
It comes down to the same issue, a rigorous definition, a repeatable study, real high-resolution audio, and the right equipment. The juries still out and I’m not sure if it really matters.
Mark, your measured response to this meta-study is a pleasure to read compared to other coverage I have seen, much of which is shallow or has a crowing, “we told you so” nature.
Look, I get it. All of the other outlets see a headline and report the highlights. They don’t investigate because they don’t know the story behind the headline. Remember how everyone took the Meyer and Moran AES paper to be proof that we can’t perceive high-resolution recordings…only to find that none of the recordings used in their study were high-resolution?
Although I don’t want to defend the M&M paper or their experimental rigour, you are way overstating what they set out to test, Mark. Your definition of high res audio didn’t exist in 2007, so they should not be mocked for using SACD-quality audio as a synonym for ‘high resolution audio’. Now, they admitted later that they didn’t know the provenance of the recordings they used, so that is a failing of their experiment. But that very fact brings up an interesting point. Let’s say that some of their test music was actually CD-grade, and some was ‘SACD-grade’. And let’s further say that some of the latter was ‘naturally’ limited for whatever reasons to no-better-than-CD, and some was not. On that basis, if we are able to hear the difference between 16/44 and ‘a little bit better’, their experimental results should have broken into two distinct sets: recordings that are distinguishable, and recordings that aren’t (which might have confused M&M, as they thought all their source music was SACD-quality). But that never happened. The fact that that never happened is very telling, I think. I hope you follow my logic.
Grant, I don’t think so. Their paper purported to show the “audibility” of CD spec sound vs “high-resolution audio” playback, which was interpreted to prove that HRA was imperceptible. They got their research published in the AES journal and a lot of attention was paid to their publication. High-resolution audio started to emerge around 2000 when the DVD-Audio format was just getting started. I took the definition as 96 kH/24-bit PCM at that point and started making new recording at those specs. My criticism of their study is based on the fact that their society members simply purchased commercially available ‘hi-res” content and played it in its original form and then downconverted (in realtime) to CD spec. Surprise…no one could tell the difference. And the reason was that the sources weren’t high-resolution to begin with…they weren’t “better” than CDs. Simply putting a sticker of logo on a recording doesn’t make it high-resolution. A high-resolution recording must be made using high-res equipment at the original session.
They knew the provenance of the recordings but never thought about acquiring “stimuli” that actually produced ultrasonics or higher dynamic range fidelity through their systems. SACD quality doesn’t mean anything…the limiting factor was the quality of the original source that was used to make the SACD.
Although I like the Meyer and Moran study, then I see your point, Mark. Although I personally feel that hi-res is pointless except for a few theoretical reasons, then I completely respect your stance on the matter, and I’m impressed about how honest you are in the debate compared to most hi-res advocates who have the typical attitude of “hi-res is so much better, and it’s so goddamn obvious, so if you can’t hear it, you’re deaf!”. I much preferred how you presented the matter in your Youtube video “high resolution audio demystified”.
Nevertheless, I like what someone else said about the Meyer and Moran study:
“THINK ABOUT IT: Two guys come along and the most respected objective audio organization in the world publishes their paper that threatens to destroy the entire SACD audio industry and also does damage to high resolution audio formats of any kind. Meyer and Moran attempted to demonstrate CD quality audio really is good enough. If Philips, Sony and the music labels behind SACD knew the study was flawed, and SACD was audibly superior, it would have been pocket change for them to fund a study demonstrating where Meyer and Moran were wrong. But, surprise surprise, that never happened.”
So, as was mentioned in a debate about the 2016 meta study, we’ve had hi-res for around 25 years, and so far nobody has been able to do a properly conducted study that shows that hi-res is audibly superior to CD quality. To me that’s very telling. Maybe we should give up fighting this battle that’s really not of much practical use :-).
Grant: If what you said is accurate , namely : “Now, they admitted later that they didn’t know the provenance of the recordings they used, so that is a failing of their experiment.
It’s beyond a failing if this shit (M&M study) actually holds some influence. If it does, it should be affectionately (and honestly) be referred to as the ‘Meyer & Moron Study”. A study of stupidity. That’s shameful. Disgraceful actually.
peter jasz
Hello Mark
I read the paper as well, and would like to thank the man for an exhaustive study of all previous studies. But one phrase that really stuck out was; sixty percent of trained listeners could hear a difference. That’s pretty close to coin flip territory if you ask me.
Robert, hitting 60% would be good enough for me. But by the time you cherry pick only 18 studies out of the whole lot and then do your meta analysis, what can you really show. The original studies are both good and not so good. We didn’t have definitive proof before and we still don’t.
The study (based on previous flawed research) was weak at best. There are many factors involved when defining precision(high-resolution) music playback.
The study (and studies he examined) was far from taking many of these factors into account. That we need “proof” lies only in the weak minds (and even weaker hearing/processing functions) by these lab-coat Phd types.
Was there anything at all insightful, useful or even accurate in any of those foolish, half-baked “studies” (that appease the science community and no one else) ?
Looking to science (as is currently understood) is a hit-miss science when it comes to defining something as intricate as music/speech reproduction -what qualifies as ‘natural’, and the reasons why. I believe Stuart’s latest MQA work may very well shed some insight into the topic, but clearly its post Reiss’s (and that what he referenced) published turd.
Once we become better acquainted with Quantum Physics will we then have a better understanding to what clearly plagues (and sets back) any meaningful understanding of current knowledge -that I assure you will provide some clear “evidence” so desperately demanded by academics (whose understanding is about as precise as the movements of a fish out of water.
It wasn’t that long ago when it was believed a capacitor was simply a capacitor (rating, tolerance -end of story) , or that a loudspeaker that ‘measures’ the same will sound the same -or an amplifier.
Our (collective) knowledge has advanced considerably the past 20-years. Now, if only academia could keep pace …
peter jasz
“Frequencies of Inaudible High-Frequency Sounds Differentially Affect Brain Activity: Positive and Negative Hypersonic Effects” paper published in 2014 by T.Oohashi (among others) offers some explanation as to why some could or could not hear the difference in the so called Hi-Rez recordings. In that paper it was found out that frequencies above 16 -32kHz actually induces negative effect and to reap the benefits (positive effect) of Hi-Rez the frequency content should extend above 48kHz. That probably explains why people couldn’t differentiate SACD which filters everything above 50kHz.
Somehow, this paper did not get much attention.
SACD usually gets filtered lower than that because the noise creeps in on a DSD 64 track well before 30 kHz.
Oohashi got a huge amount of attention and was eventually discredited. Not overlooked at all. At least two decent attempts were made to confirm his ‘hypersonic effect’ but they only got no such effect.
That’s unfortunately often the problem with many studies. Either they get discredited, or it’s not repeatable.
Arnie Krueger mentioned that he did a study in the late 70s, which yielded a positive result for something they didn’t expect a positive result for. Then they tried to repeat the exact study with the exact same subjects, and they got a negative result. So, as it’s often arbitrary, you sometimes get a positive result purely by chance, but it’s not repeatable.
Just an analogy: I’m from Denmark. We’re only 5.5 million people. Just the other day a shop selling lottery tickets sold their EIGHTH jackpot ticket in the 20 years they had been in business. Obviously, no cheating was involved, but it’s rare for such an occurence in such a small country. Statistically speaking it’s probably highly unlikely. Another story: One of my first ABX tests was for Pink Floyd’s “Dark side of the moon” SACD vs. the 2011 remaster if I’m not mistaken. I got 2 out of 8 correct. One study took an example like that from their study (people gave the wrong answer) and concluded that they simply “pressed the wrong button”, so people could hear the difference between hi-res and CD quality. During my ABX test I really thought they sounded completely identical, and I just guessed, although statistically speaking I should have had 4 out of 8 correct. Even if I flipped a coin 10,000 times, I probably wouldn’t get 5,000 head and 5,000 tails. Maybe it would be 4,000 head and 6,000 tails.
In another ABX test (different music) I just pressed A for every trial without listening and got 9 out of 12 correct. Then I tried this approach again 20 times. Once I got 8/12 correct and once 7/12, but the rest of the times, whether it was 12 or 20 trials, I got no more than 50 % correct. The first ABX with that material I got 4 out of 12 correct, as I really couldn’t hear any difference. Other ABX tests I have passed with 20 out of 20 correct, and I could repeat them. ABX tests with less than 12 or 16 trials are usually less reliable.
Personally, I’m more in favor of filtering out supersonic frequencies than preserving them. I couldn’t imagine that the screechiest sounds imaginable would be a pleasure to listen to, nor could I imagine that any of the sounds in that frequency band would be worth listening to – insect noise, noise from storage units in the studio, buzz, hum, etc., although I am aware that certain instruments (like cymbals) produce tones in those frequencies. On top of this, there’s the possibility of intermodulation distortion. But that’s my personal opinion, and I know Mark disagrees, and I’m fine with that :-).
With all due respect, 48/20 *is* high-resolution audio, and not just in the minds of the previously mentioned organizations. It may not be the highest available, and maybe there needs to be multiple classifications for HRA, but that spec meets the criteria of being more than 44/16. Surely you’d concede that all else being equal, a 20-bit recording should sound better than 16-bit?
I disagree and I realize I’m in the minority with my more rigorous definition. But it really doesn’t matter anyway because none of the music releases come anywhere close to using the available fidelity provided by the CD spec. The Reiss paper specifically ignores word length. Why? Because you can’t find a single mainstream commercial release that uses even 16-bits…so why move higher. The studies focused on the sample rate…moving to 48 or 96 or 192 kHz. The DEG, CEA, NARAS, and labels want a low standard for High-Res Music because it allows them to perpetuate the myth that older recording newly transferred into high-res bit buckets will have higher fidelity. It’s not true. High-res has the potential to be better than CDs, but virtually none of the stuff that is called high-resoluion audio actually is.
Hi Mark
Apart from the time, what is stopping you from conducting a listening test. I appreciate that this is not a full study, but you are in a position to sit a group down and play Redbook versions and Hi Res versions of the same tracks.
I am certain that you would get something worth reporting on if you could over, say, three sessions have 15 people (5 per session) do the same test. I say “test” as I don’t know what else in terms of preparation has to be done beforehand to ensure validity. But it would be very interesting to see if your version (which I whole heartedly believe and support) Hi Res is perceptible to the groups.
When I saw the words “when test subjects received extensive training” I wonder what that training might be and if that training created a bias of some kind.
I can only imagine that if you give a group an example of the difference before they start the test, you create a bias. However as I am not a professor or the like I am somewhat naive in some of my statements. But I think you get my meaning.
When you invited some people to attend a workshop at your studio last year, they got to listen to real Hi Res from your catalogue, did you play any redbook at those sessions? (Just wondering)
What do you think of the idea of doing a study yourself, not so much in disproving others but proving your side of the subject, not that you need to prove it, just that a study that starts with the correct premise for Hi Res audio has a better chance in the first place of delivering the answer = Hi Res transfer are indistinguishable from their redbook counterparts true Hi Res is distinguishable.
If the industry can not understand that if the source is not Hi Res the output, however treated, is still not Hi Res, then more fool them, a well conducted test/study by someone with the necessary understanding in the first place would make interesting reading and equally interesting results.
Still find it funny that this guy’s study, is a study of studies based on an incorrect understanding of what Hi Res is, and then one of the biggest most experienced people in the industry applauds it. Anyway, thanks for the blog, we will get it right one day.
Gordon
I could do this study and came dangerously close some years ago. I would like to secure a setup as described in the paper that actually delivers ultrasonics with greater than 120 dB of dynamics and play my recordings at Redbook and high-res. But I’ve come to realize that even if I did the work, it doesn’t really matter since the artists, recording industry, labels, and CE companies don’t really care. They love the hype and the marketing possibilities but when it comes to really incredible fidelity…meh.
” I would like to secure a setup as described in the paper that actually delivers ultrasonics with greater than 120 dB of dynamics and play my recordings at Redbook and high-res.”
Who would argue no difference there?
You have recordings with peaks 120db above the noise floor?
The dynamic range of the recordings can exceed the limits of CD specs and be as good as the venue where the recordings were made. And the playback environment hast o be better than that to get receive the benefits of the increased dynamic range. Yes, I have recordings with greater than CD spec. To do this sort of testing would mean evaluating the very best case…thus far, it hasn’t been done.
“The dynamic range of the recordings can exceed the limits of CD specs and be as good as the venue where the recordings were made.”
Right, that’s never been in question. The questions are, from what audience perspective and relevance to consumer playback systems? You’ve answered the latter. 😉
“And the playback environment hast o be better than that to get receive the benefits of the increased dynamic range. Yes, I have recordings with greater than CD spec. To do this sort of testing would mean evaluating the very best case…thus far, it hasn’t been done.”
How many specific recordings like this do you have and are they available for purchase? I have systems capable of that dynamic range with hypersonic capability, though I’m not personally a believer. But have access to plenty believers, so can conduct my own investigations.
I was simply unaware that for the purported benefits of Hi Rez to be demonstrable, both conditions needed to be met simultaneously.
Many of my recordings have very high dynamic range and they all have greater than 20 kHz frequency response. I would start with the FREE samples that I provide through the FTP site. I know “Mosaic” by Laurence Juber is a wonderful example. Check it out…I’ll be curious what you find.
Ok, thanks Mark, sent FTP email.
Now just to be clear, you are not suggesting that there is audience position capture of >20khz signals at 120db + peaks, correct?
It’s funny because just today I’m working on the illustrations for the book and created the “inverse square rule” illustration. Of course, the distance to an audience plays a large role. I’m want to be standing next to the conductor or on stage with the band. I simply want our recordings and reproduction systems to be capable of handling real world audio…without compromise. If producers or artists or anyone else wants to change from that ideal to something else because of a creative need…so be it. We can do this now…so why not?
I filled in the online form 7/3 to access the “FTP site”, but have not received a response.
Hard to say whether it didn’t go through, was spammed or is related to your recent surgery….
I sent the link via PM.
It is extremely difficult to devise experiments where only the one variable under test is the only variable in the experiment. A careful look at the methodology must conclude that there are no other possible explanations for the results. In one famous experiment I’d read about, a professor of electrical engineering in Japan tried to demonstrate to his students in a careful experiment that sounds above 20 kHz are inaudible. When his students heard a difference, he re-examined his procedure. It seems energy above 20 kHz was producing distortion in the tweeter below 20 kHz. The problem was resolved by using a separate tweeter for sounds above 20 kHz and cutting off the response of the normal tweeter above 20 kHz. Then the expected results were obtained. Even the most objective and knowledgeable people can make mistakes that lead to the wrong conclusion.
I DUNNO EVERYTHING (EVEN SOCRATES DIDN’T) , BUT IF WE TAKE CD’S HUMAN PERCEPTIBLE FREQUENCY SPECIFICATION (2x22050Hz) AND PUT DATA BITS ENOUGH TO REACH ANALOG DATA INFORMATION, COULDN’T WE HAVE THE PERFECT AUDIO FORMAT??? (+TAGS, IMAGES ETC.)???
(JUST MY OPINION).
There’s a lot more to actually engineering high fidelity than just the numbers. Filter cutoffs etc.
The first problem is to define exactly what the term “high fidelity” means. It is here that I part company with Dr. Waldrep.
Paraphrasing Robert Frost;
“Two roads diverged in a wood, and I—
I took the one “NEVER” traveled by,
And that has made all the difference. “
I have been reading about this absurd debate for so long that its continued existence is becoming an embarrassment to the industry. Such an intelligent group of people not being able to muster the voice to settle such a trivial- yet personally significant- matter is beyond folly.
I do understand that there are some relatively massive financial interests actively injecting misinformation into the public sphere; the laughable “experiment” conducted during Pogue’s review of the Pono is one obvious recent example. But, the facts are ultimately on our side.
I cannot count how many times I have heard/read about some new product bringing HiFi to the masses… good luck with that if they don’t believe HiFi actually exists. These proponents are losing to a propaganda 101 playbook.
Hi Nick: What ‘side’ of the debate do you subscribe ?
Great reply though !
peter jasz
http://www.PONOSUCKS.com
I’m a huge fan of Neil Young as well…but your points are well taken.
Hallo,
I know that most of you do not understand German.
Nevertheless the is a study which shows how to compare different recording techniques.
It was done as a work for a diploma at the Detmold Academy of Music – Erich Thienhaus Institute in Detmold Germany.
Here you will find ALL information on the conditions and the test setup.
Maybe you’ll try the translator function of Google to understand how the study was made and the conclusion on the tests.
http://old.hfm-detmold.de/eti/projekte/diplomarbeiten/2004/dsdpcm/xdslindex.htm
The study was made in 2004 and is still valid. It shows one more time that there is NO need for DSD.
Regards
My German is not very good although I did study enough to pass reading proficiency for my doctoral exams. I’m afraid my abilities won’t get me through a technical paper but I will try to get to the conclusions in this paper.