NPR Makes A Splash With Audio Quality Comparison
On June 2, Tyler Fisher and Jacob Ganz asked a very real question on the NPR Music website “How Well Can You Hear Audio Quality?” If you haven’t taken the test, head over to NPR.org and check it out. The authors selected 6 different music selections, extracted a short (10-20″) excerpt, encoded them at 128 or 320 kbps in the MP3 format, and left them in the “uncompressed .WAV format”. Users can play each of the excerpts and then choose which one is the highest quality.
The introductory paragraph provides the catalyst for the quality quiz:
“Recently, the rapper Jay Z relaunched the subscription streaming music service Tidal, which includes the option to listen to high-definition audio for $19.99 per month. Tidal’s HiFi, with its uncompressed audio files, promises a better listening experience than any other streaming service on the market.”
The claim that Jay Z’s TIDAL site is offering “high-definition audio” streams is wrong on two counts…and that’s only the first sentence. A brand as respected as NPR should refer to “high-resolution audio” NOT “high-definition audio” and they should know that streaming 1411 kbps CD specification audio is not high-res, even by the lame standards insisted on by the DEG, CEA, NARAS, JAS, and major labels. The writers didn’t get off to the best start…and there are other problems, as we’ll see.
The actual quiz is lacking in a number of important ways. The most significant is the fidelity of the recordings that are offered. As you’ll see shortly, most of the tracks are heavily mastered and none of the uncompressed .WAV files qualify as HiFi quality. And then there’s the problem of presentation. When I do comparisons in my studio, I always allow the person being tested to instantly and noiselessly switch between the two options. Most of the time these test are performed…and in the present NPR case…you get to listen to a selection from the beginning and then listen to another choice “from the beginning”. This means you have to rely on your “musical memory” instead of a direct comparison. It is really hard to succeed at a test like this.
I got 3 out of 6 correct sitting here at my computer listening through my Oppo PM-1 Planar Dynamic headphones from a Benchmark DAC2 DAC…I had a very good signal path!
The order of the tracks and formats are randomized so you can’t game the test. The first example that came up was the classical selection by Murray Perahia & The English Chamber Orchestra.
Figure 1 – The NPR hearing quiz Murray Perahia selection and accompanying text.
I read the accompanying text and was surprised to see this was “released in 1981” because the first commercial disc wasn’t produced until a year later. Billy Joel’s 52nd Street with the first album to be released on CD along with the Sony CDP-101 CD Player (which came out on October 1…thus the 101!).
I missed this one. When I looked at the spectrogram (I recorded all of the examples following the test), I didn’t feel too bad. Take a look at the spectra below:
Figure 2 – The spectrogram of the Murray Perahia selection on the NPR test. [Click to enlarge]
This was a very early digital recording intended for the new CD format. The recording was done at 44.1 kHz and 16-bits. The music is pretty quiet and not highly dynamic. The excerpt doesn’t contain much musical information above 10-15 kHz. And the information that is up in that register is down 85 dB below the peaks. This means that a person would have to be able to hear very subtle musical harmonics 85 dB below the normal level of the music in order to get this one correct. There are other subtle distortions contained in the MP3 files but a 320 kbps encode fooled me.
The MP3 files have no high frequencies above 16 kHz…and the uncompressed WAV files has energy up there but no music. Not the best piece to test with IMHO.
I’ll dissect the other 5 tracks tomorrow, so stay tuned. I’d love to hear how you did and the equipment you used when taking the test.
I used a Mac Mini with Bose AE2 headphones plugged directly into the headphone port. I rarely do well on these types of test with this combination and, indeed, I only got 2 out of 6. That’s essentially just guessing.
I did find it notable, though, that I could hear differences between the tracks in a few cases! This is where my lack of training comes in, as I often couldn’t tell which sounds were supposed to be there and which weren’t. In particular, one of the songs (forgetting which one) had an irritating high frequency repetitive sound (very harsh) on two of the tracks and not on the third. I assumed that that must be MP3 noise and chose the “quiet” one. Oops… that one was the 128kbs track! It seems that that sound was supposed to be there and it was the 128kbps version that didn’t encode it.
Going back over them again, I realized that I could never pick out the difference between CD quality and 320kbps MP3 with my setup, but I could occasionally pick out the 128kbps MP3.
Kurt, this is very challenging…and anyone that thinks it’s not is lying.
I got 2 of 6 right. But I was always able to tell between the lower bit rate mp3 and 320kbps. Each one I got wrong, I got wrong because I chose the 320 mp3 version instead of uncompressed WAV. My equipment chain: MacBook Pro > Schiit Audio Modi 2 Uber DAC > Magni 2 Uber headphone amp > Grado PS500.
Bravo! I noticed exactly the same things you did. It’s a rediculous and pointless non-“test”, proving nothing, and there’s not a shred of high res in it. NPR? You should be ashamed!
BTW, I go 3 out of 6 also. Same 3.
To be fair, the NPR writers didn’t claim that the uncompressed files were “high-definition”. But they made very bad choices in music to test.
Hi Mark,
The fact that an institution like NPR prepared such a misleading text and test, is highly disappointing. I thought they would have enough people capable of informing themselves better before putting up this mockery of a listening test, and take their readers – and their prestige – a bit more seriously. Instead we have yet another force getting behind the hype and the outright BS that HRA has been turned into. They sure deserve a letter from you, after which I would expect them to correct themselves; and since they have the audience – my FB was plagued with posts of this test and tons of misinformed comments – set up a better test that actually tests something instead of selling TIDAL.
I would have used my Benchmark DAC2 and my Sennheiser HD 800s for the test, but after I saw the tracks and googled some info on the recordings for some provenance data, I let it go.
Cheers!
To be fair, the NPR writers didn’t claim that the uncompressed files were “high-definition”. But they made very bad choices in music to test. The whole area of HRA and hearing between MP3s and CDs is challenging. I’m not really surprised the NPR got it so wrong…but I am disappointed that they didn’t seek out an authority.
I also got 3 out of 6 right. The ones I got wrong, I had picked the 320 kbps MP3 as the best sound and one of them was the Murray Perahia song. I also got Coldplay and Neil Young wrong. The ones I got right sounded more open with a wider soundstage. There was this extra bit of detail. Ofcourse, it is a difficult test and I had to listen very carefully (a very quiet room, eyes closed, NAD Viso HP 50, Oppo HA – 2 from a laptop). Since I picked 320 kbps and not 128 kbps for the songs I got wrong, I’d like to believe I wasn’t guessing !
Here’s my results, and after 6 runs, I’m perfect on Katy Perry:
https://docs.google.com/spreadsheets/d/1D_LWAJfTZIbrfd7x5llQG7nVxZ6g-LYnLWIoZdsB5Pk/pubhtml
I’m shooting to do 10 trials. Only 1 can’t be better than random guessing, as proven by the first column where I used a die to make my choices. The other hard part is familiarity. I was only familiar with 1 of these tracks in depth, and just because you hear a difference, which I regularly do in all but 2 of the tracks (the Neil Young track is, ironically, the worst track, IMHO, for hearing the difference between 128 and uncompressed), doesn’t mean that you’ll correctly attribute that difference to the proper bitrate. So far, my ability to get a perfect 6 seems to be going up with time. I’m fairly confident that in something close to an ABX, I could get 5/6 or better every time. After my 10 trials, I do a couple runs with one window and the tracks completed and identified, and another window blind. By switching between the known lossless track and the unknowns, I’m pretty sure that I’ll do well.
PS, you can click in the middle of a track to skip to a specific part. Not quite instantaneous switching, but close.
Equipment is mid grade: Line 6 UX2 to Senn HD 280 Pro. I’ve also done one run with that same DAC to a Behringer Mixer through Behringer Truth B2031 Monitors.
Thanks for the information. Read today’s post…you’ll find it interesting. The fact that the MP3s at 128 are slightly louder is curious, for example.
3 out of 6 just happens to be spot on the statistical average for random selection lol lol
Well not really, when there are three choices from which to choose. For A|B yes, but getting 3 of 6 right is better than 50/50.
‘I read the accompanying text and was surprised to see this was “released in 1981″ because the first commercial disc wasn’t produced until a year later. Billy Joel’s 52nd Street with the first album to be released on CD along with the Sony CDP-101 CD Player (which came out on October 1…thus the 101!).’
Mark, as you are surely aware of, there where digitally recorded vinyl LPs (mainly classical) which were released much earlier than the first CDs. I happen to own quite a few of them all of them from DG. And yes, they do exhibit characteristics which point to the inadequate understanding of the digital process on the part of recording, mastering and re-mastering engineers of the time.
And btw, R. Strauss’ EIne Alpensymphonie with the Berliner Phil. conducted by Herbie was the first commercial CD produced and manufactured (digitally recorded in 1980 afaik) but not released until somewhat later. Comparison between that release and the re-mastered release of 1993 (DG Gold series) shows that most of the problems were not with the recordings but with the mastering or transfers be them for CD or ‘analog’ vinyl.
I did know about the Billy Joel album…I knew the engineer. The Wikipedia article mentions one classical release in 1981 and an ABBA CD overseas so I’m not sure about the date on the Perahia CD…But I doubt it was 1981.
2 out of 6 right. Used an Audioengine D1 external DAC plugged with an Audioengine N22 amp driving two passive Audioengine P4 speakers. AT my normal listening levels I could not tell much difference it was quite hard. If I really cranked the system, to the point that I thought I might blow the speakers, I could hear some difference but still found it hard and never ever ever would listen to anything that loud anymore.
I don’t have the liner notes, but the front cover for the Perahia recording includes the text “Digitally Mastered Analog Recording” in the lower right corner.
Thanks, that’s helpful.
Hi Mark, great post as always. My colleagues and I at Harman’s engineering center in Shelton, CT, listened to this material via Mac USB asynchronous streaming > Mark Levinson No585 integrated amplifier’s DAC > JBL Everest loudspeakers with high-end cables and a dedicated power source. We could not come to any consensus whatsoever, though we did not listen super loud. Nobody really seemed to get more than two or three right. Very frustrating portrayal of hi-res audio!
Thanks Dan. Not surprising…it’s very difficult for a number of reasons.
I got 5/6 correct, using Logitech computer speakers (at work, so they were probably the cheaper ones). I got the Katy Perry one wrong.
Very good!
“The excerpt doesn’t contain much musical information above 10-15 kHz. And the information that is up in that register is down 85 dB below the peaks. This means that a person would have to be able to hear very subtle musical harmonics 85 dB below the normal level of the music in order to get this one correct.”
Thanks for writing that, Mark. This is a point I’ve made for years and one that’s rarely mentioned by audiophiles [sic]. A spectrograph might show energy way up in the ultrasonic area which is easily VISIBLE to the eye, but high harmonics are ALWAYS accompanied by vastly stronger low frequencies that thus mask the harmonics so that the EARS cannot detect them. A well-produced multichannel 5.0 Blu-Ray or SACD recorded at 44/16 will bring much more listening pleasure than a well-produced 384/32 stereo download.
Come on man you’re just butthurt. NPR’s is a perfectly reasonable and interesting article.
They’re talking about Jay-Z’s Tidal service and so asking how much Jay-Z’s music benefits from lossless. Course it benefits less than something with a serious dynamic range but it certainly benefits. (I find a lot of electronic music with sawtooth or square waves benefits most as they’re not efficiently represented by a fourier transform looking for sine waves)
I too, consider myself to have good ears and also got only got 3 right. In fact I felt it to be a bit like retraining my ears. I got the first 3 wrong, was reminded by those of what I was meant to be listening for and then got the rest right. There are many musicians in the comments on the article claiming to have got 100% and I believe them. The difference is there and meaningful , even in these genres, if less so than for those with dynamic range.
Fair enough, aural memory comes into it and getting the wrong answer when overthinking it doesn’t means the higher quality version isn’t enhancing your enjoyment.
Also, why did you use a spectrograph to tell you what the difference was?! Getting it right in the gain/frequency domain is exactly what mp3 compression is good at. It’s the time domain, the animation of that frequency profile over the milliseconds which is hard for fourier transformed compression to get spot on.
I did the breakdown of the tracks to illustrate the differences and the similarities.