HD-Audio Challenge Part II
Thus far I’ve received 83 responses to the HD-Audio challenge — a pretty good number for a casual survey. I spent a couple of hours going through them and created a spreadsheet. Please keep in mind that I’m not a statistician and am ill-equipped to do any fancy data analysis on the responses. But it is obvious to me that some questions can be tentatively answered.
Here’s a few:
1. How many people got 6 correct? 2
2. How many people got 5 correct? 10
3. How many people got 4 correct? 13
4. How many people got 3 correct? 13
5. How many people got 2 correct? 18
6. How many people got 1 correct? 6
7. How many people got 0 correct or couldn’t tell any difference? 17
I did receive a number of comments/emails after posting the results. As I expected, some people blame me for their inability to pick the high-resolution option. Here’s my favorite:
“Just listened to the tracks and wanted to share some thoughts…
All the files are 24/96. Did you downconvert to 44.1/16 and then upconvert? Seems like it would have been a more valid test to rip a CD with EAC and then compare to a 24/96 version. Or download the different resolution files from a commercial service. Less of a chance for conversion artifacts to be introduced into the files. What was the source of the original 24/96 files? There wasn’t anything in the metadata to indicate their source (HDtracks, etc). Hopefully not ripped from vinyl and then resampled.
As you know the quality of the original recording and mastering ultimately determines the potential fidelity of the sound in any file format. I listen to a lot of music in DSD format (and 24/96-192 PCM) and none of the recordings in your test were as good as the best engineered music available today (both analog masters from the 60’s and 70’s and current digital masters (sans compression). They all lacked the depth, presence (soundstage) and detail available in the best engineered recordings. If you don’t start with high quality masters then any comparison is specious.
Thanks for creating the challenge, my suggestions for the next time you do this is to compare DSD versions with a direct CD rip (or redbook audio file from the source) using high fidelity mastered content (e.g. 2xHD). I’ve done this comparison, the difference in the sound quality (detail, soundstage/presence and depth) is amazing.
XXX
p.s. I got 5 of the 6 comparisons wrong so there’s something in the way you’ve created and resampled the files (in addition to the mediocre fidelity of the original masters) that is introducing variability in the responses. I noticed that some of these were 5 years old and edited with Audition.”
I replied:
“Thanks for participating. These files came from my award-winning catalog of native 96 kHz/24-bit PCM masters. As I explained in the original blog post, my original master files were downconverted from 96/24 to CD res and then placed in a 96/24 container to make the file sizes the same. Using content downloaded from a commercial download site is always problematic because virtually all of the files do not exhibit better than CD fidelity. The methodology used in this survey guaranteed both files have the same provenance. The only difference between the original and CD version is the sample rate and the word length. This makes this a uniquely valid comparison.
The sound or “fidelity” of the music you enjoy may be preferable to your ears through your system but the actual fidelity is likely very limited. The heavy use of compression and tweaking of EQ (things that I don’t do) create the usual commercial sound, which lacks dynamics and extended frequency response. What you describe as lacking “depth, presence and detail” are actually attributes that make these recordings exceptional—and real high-resolution masters. You obviously have different listening tastes than those that cherish my tracks. AIX Recordings have won numerous awards and glowing reviews for almost two decades. I acknowledge that they do have a very different sound than most releases.
There would be too many problems using a DSD master and a ripped CD…they don’t come through the same production processes. I’m not a fan of DSD nor 2xHD recordings. What you hear is something other than the difference in resolution. It should be somewhat telling that you were unable to perceive a difference between the same master in its original resolution and in CD spec. I created the catalog over the past 18 years…all were done with the same equipment, resolution, and mixed in the same room.
Thanks for your comments.
Mark
PS It turns out that very few (less than 5 out of 100) people that took the survey were able to identify 6 out of 6. Many responded that they couldn’t perceive any differences. Like you, I was hoping that using high-resolution would be perceptible.
I’ll talk about the ramifications and some additional details in a future post.
IMO using DSD for resolution comparison test is problematic. I can tell difference between DSD and PCM. Apparently DSD encoding or decoding or both introduces some coloring that does make it sound less natural. I have the same problem with vinyl records due to mechanical nature or reproduction. I personally prefer transparent and neutral sound. One of downsides of CAS I attended in July was that most systems for some reason played back vinyl records which made difficult to estimate actual sound quality.
If you can tell the difference between a DSD and PCM recording that came from the same source (microphones of analog tapes) then you’re better than the people that participated in a study of exactly that. The researchers determined that no one could tell the difference in that study. High-resolution PCM is simply the format of choice because engineers can work with it and it produces “transparent and neutral sound”. Trade show vendors (and magazines/websites) like to hype the latest trends regardless of whether they deliver improved fidelity of not. Take vinyl, MQA, DSD etc. as examples.
No need to publish this, but I only get 79 when I add up the responses. Might want to check the count again.
There were some that had a mixed bag of “no choice” and selections. I only counted those that were complete.
The reason why your recordings in a CD resolution sound so good obvisiously is, that they were originally made in Highres (24/96).
This just shows that good stuff will keep on sounding good even in 16/44.1!
Exactly the point of recording using high-resolution during production.
As a matter of pure chance, if the files were truly indistinguishable, you would expect about half of the folks to be able to guess half of the six pairs correctly. Adding up the 79 (not 83) responses in your list, 38 folks were able to guess 3 or more correctly, and 41 folks were able to guess less than three correctly. I’d say that’s within the margin of error.
I was one of those who couldn’t hear any real difference. I initially put it down to doing this with midrange quality headphones (AKG K240 Studios – the current version of the phones you’ll see musicians wearing on old album covers – the ones with two black spring wires arching high over their heads) on a Dell laptop with a soundcard claiming 96/24 quality – or to my being 68 2/3 years old – though I’ve always had better than average high frequency hearing (I had to trade in my old Yamaha electric piano for a Casio because the Yamaha didn’t have the overtones fade out – too little chip memory for that – so they simply recirculated as long as I held down the sustain pedal, causing a traffic jam in the hight frequencies), but unless the whole test population is Boomers like me, I think this test is pretty conclusive.
It’s still worth buying AIX’s records because of the musical and production quality and “stage perspective” surround mixes, however.
I don’t begrudge some commercial records their compressed dynamic range – like Steely Dan’s “infinite mix” approach that lets you hear every layer clearly – it’s like the “heightened reality” of sci-fi thrillers. (The DVD-A of “Gaucho” – also a stage perspective mix – is a masterpiece.) Sometimes you just want something a bit more exciting than everyday life. That’s why the Motown performers wore sequin-covered clothing – so they’d glitter in the spotlights.
Thanks Phil. I enjoy the Steely Dan and other highly produced and tastefully mastered albums. It important to recognize that those tracks use only very few bits, however.
Hi Mark,
Regardless of the high definition debate you have confirmed how many people are quite prepared to show you they haven’t got a clue what they’re talking about….
Good morning John. As I’ve stated repeatedly, the music industry is a business. I saw that during my time on the CEA audio board. The group of manufacturers — and the organization — has absolutely no interest in the truth or fidelity. They only cared about maximizing profits for their members. But that shouldn’t come as a surprise.
“How many people got 0 correct or couldn’t tell any difference? 17”
Could you split that out into how many tried and scored 0 versus how many said “no difference that they could hear”?
Many thanks.
There were 17 individuals that replied “no choice” for all 6 selections. Then there were a number of submissions that had one or two selections and the rest “NC”.
You really shouldn’t group the no choice with the 0 of 6. Because had this been a forced choice you would expect the no choice people to average 3 of 6 correct. No difference/no choice isn’t the same as scoring zero. It would equate with getting half right.
In any case your results are essentially random.
If I’ve done the maths correctly, the statistical outcome if people were guessing would be as follows, with deviation of the actual outcome indicated:
6: 1.6%, 1.2/79, +0.8
5: 9.4%, 7.4/79, +2.6
4: 23%, 19/79, -6
3: 31%, 25/79, -12
2: 23%, 19/79, -1
1: 9.4%, 7.4/79, -1.4
0: 1.6%, 1.2/79, ?
Since 6 incorrect responses are not reported separately from those who did not make a selection, we can assume that had they simply guessed, they would have followed the statistical distribution. These 17 entries should thus be distributed as 0.3, 1.6, 4.0, 5.3, 4.0, 1.6, 0.3. This gives an amended tally of 2.3, 11.6, 17, 18.3, 22, 7.6, 0.3. The deviation from the expected value here is +1.1, +4.2, -2, -6.7, +3, +0.2, -0.9. There is a shift towards correct responses, though there is also an elevated number of 2 correct. Given the sample size, deviations of this magnitude are probably normal.
My conclusion is that at most 16 people were honest enough (with themselves) to say they couldn’t hear a difference.
Thanks for this additional analysis.
I went back and took another look. I’m not sure how to handle people the had “NC” on some a a couple of selections. There were 17 people that picked “NC” for all of them.
If nobody replied with all wrong, it’s better to leave those 17 out entirely. The expected distribution of the remaining 62 then becomes:
6: 0.97, +1.0
5: 5.8, +4.2
4: 15, -2
3: 19, -6
2: 15, +3
1: 5.8, +0.2
0: 0.97, -0.97
There are many ways to spin this. For instance, one might say that the number of people (12) getting at least 5 correct exceeded the expected value (6.8) by 76%. Clear proof that high-res is audible! However, if we look at how many got 3 or more correct, the number falls short of the expected by 6.6%. High-res is obviously not worthwhile! Statistics, the liar’s best friend.
I’m interested in the disparity between the number of people downloading the files but not then providing a response. My feeling is many did believing that it would be a simple process then finding embarrassment at not being able to discern any difference – possibly with very expensive gear?
I’ll try to nudge those that tried but didn’t respond.
Having put up listening tests on large forums a few times, you never get much response. The response here is actually huge compared to most. I’ve had files that get downloaded several hundred times, and usually there are between 1 and 2 dozen responses. I’ve seen similar results from others who post such files.
While it could be from a variety of reasons I tend to think it is because people try them and don’t hear the differences they expected to hear. The higher responses have been when I offer some degradation of files in several steps and at levels where some of the files are obviously different to anyone. Even then responses seem to come in rather quickly at first before stopping and few in total respond.
I have a feeling people are simply not willing to admit they can’t tell the difference because they don’t want to feel inadequate in some way particularly if they have invested heavily in kit. I can’t tell the difference but as I’ve commented before I definitely believe that the higher sampling rates have an ‘ease’ to them if you are listening for prolonged periods of time – this plus the engineering advantages Mark has mentioned many times still makes me an advocate for Hi definition recording and playback.
I got only one right – 2! Still I was not sure if I can hear the difference and most likely I was just guessing.
As promised, I tried an ABX test.
Here’s the log from the first song:
foo_abx 2.0 report
foobar2000 v1.3.7
2018-08-21 21:27:30
File A: Tune_1_A.wav
SHA1: a172a18acd31bde18e70254654eb3d6a62a98869
File B: Tune_1_B.wav
SHA1: 23140e6544890298339f2a1de731f972a0285283
Output:
DS : Højttalere (CA USB Audio)
Crossfading: YES
21:27:30 : Test started.
21:38:15 : 00/01
21:39:02 : 00/02
21:40:15 : 00/03
21:41:35 : 01/04
21:45:51 : 02/05
21:47:17 : 02/06
21:48:20 : 03/07
21:49:10 : 03/08
22:00:04 : 03/09
22:01:08 : 04/10
22:02:23 : 04/11
22:04:27 : 05/12
22:06:39 : 06/13
22:07:30 : 06/14
22:08:32 : 06/15
22:09:46 : 07/16
22:09:46 : Test finished.
———-
Total: 7/16
Probability that you were guessing: 77.3%
— signature —
215bc93fd1e452bd483b437ca134ee648c1c035a
And here’s the log from the third song:
foo_abx 2.0 report
foobar2000 v1.3.7
2018-08-22 20:11:00
File A: Tune_3_A.wav
SHA1: e4db0c5771607dd8cf1a2b629cb7dcbff593ef37
File B: Tune_3_B.wav
SHA1: 990af2551ff3a9fa1a6af0ed5b1773705464866d
Output:
DS : Højttalere (CA USB Audio)
Crossfading: YES
20:11:00 : Test started.
20:15:29 : 00/01
20:17:03 : 00/02
20:18:47 : 00/03
20:20:18 : 00/04
20:22:19 : 01/05
20:23:38 : 02/06
20:24:58 : 03/07
20:26:42 : 04/08
20:28:12 : 05/09
20:29:28 : 06/10
20:31:01 : 06/11
20:32:19 : 06/12
20:33:36 : 06/13
20:35:45 : 06/14
20:36:48 : 06/15
20:39:45 : 07/16
20:39:45 : Test finished.
———-
Total: 7/16
Probability that you were guessing: 77.3%
— signature —
b38c3f450663f07a9cd2879b9ac225faf53cb198
I had actually planned to first do an ABX test of all six songs through my speakers and then afterwards through headphones, but in the end I only ABX’ed these two songs through my speakers, as I actually thought I could hear a small difference between the two files on those particular songs. For the four others I couldn’t hear a difference, so I didn’t even try to ABX them. But as you can see from my results, I couldn’t hear a difference between the two songs I ABX’ed either.
I think it’s likely that I heard some volume level difference in the recording between A and B that seemed like a difference in sound quality, meaning at, say, 1:20 the musician was a little bit closer to the microphone, or that particular part of the song was a little bit louder than it was at 1:30 when I switched from A to B. So the “difference” I heard was in both files – in the recording and not in the resolution. I hope this makes sense.
So, now the number of people who couldn’t hear a difference has gone up to 18.
But Mark, I have to compliment you for being honest about all of this and essentially saying “well, I hoped people would be able to hear a difference, but maybe I was wrong”.
Myself and 3 other audio friends all came up with results which pretty well indicated that our selections were as good as random, whilst 3 of the 4 of us selected 4 correctly none of us agreed with each other in our results so statistically I believe this would classify as random.
However what I would like to add to the discussion is that I added an extra question to the mix and that was ; Do you think that the two files actually sound different from each other ? regardless of whether we could pick the Hi Res version or not ?
Overwhelmingly we agreed that the files sounded different to each other with 3 of the 4 of us thinking that perhaps only 1 of the 6 may have been the same file ( i.e. it sounds the same ). By the way we did this blind from each other so we could not influence each other and we just wrote down our results without discussing them until we finished.
My conclusion from this is that there is a sufficient difference in sound that we can perceive that there is a difference between CD and Hi Res , we just can’t tell which one is Hi Res !
Maybe it is just that our sound systems are not good enough to allow us to discern ?
I wonder how many other people thought the files sound different to each other and how many thought that they sounded the same ?
Sort of a technical question. If the output of the computer is set to 96/24 and each of the versions is played with that setting. Assuming bitperfect transmission of the file to the DAC, will the simply ignore the filler on the lower res file?
The sample rate of both files is identical.