I finally found some time to do the ABX tests and write up a discussion as to their relevance.

Results

Bars in red are NOT statistically significant (p > 0.05).

Bars in blue ARE statistically significant (p < 0.05).

Figure 1: ABX test results for the sample Berlin_drug showing the percentage of correct differentiations of a FLAC version of the clip when compared to various bit rate MP3s.

Figure 2: ABX test results for the sample Enchanting_sample showing the percentage of correct differentiations of a FLAC version of the clip when compared to various bit rate MP3s.

Figure 3: ABX test results for the sample Finalfantasy showing the percentage of correct differentiations of a FLAC version of the clip when compared to various bit rate MP3s.

Figure 4: ABX test results for the sample Human_Disease showing the percentage of correct differentiations of a FLAC version of the clip when compared to various bit rate MP3s.

Figure 5: ABX test results for the sample King of the Mountain showing the percentage of correct differentiations of a FLAC version of the clip when compared to various bit rate MP3s.

Figure 6: ABX test results for the sample Nautilus showing the percentage of correct differentiations of a FLAC version of the clip when compared to various bit rate MP3s.

Figure 7: ABX test results for the sample WelcomeToDrexciya showing the percentage of correct differentiations of a FLAC version of the clip when compared to various bit rate MP3s.

Raw Data

I am a great believer in showing your work, so I have uploaded my raw data, as it be, as further evidence of my results. This is in the form of an Excel spreadsheet I entered my results into and made graphs in, as well as the output from Foobar for each of the samples.

1. Excel Spreadsheet

2. Foobar Output

Discussion

Right from the outset I knew that regardless of my results I would still store my music in the Apple Lossless format for several reasons I have mentioned previously. By using lossless I am essentially getting the full experience with no loss of quality whatsoever. Face it, technology evolves, we don’t know where it will go, so having the original CD quality can only be a good thing. Likewise, my hearing may become more finely tuned to hearing differences between lossy and lossless audio so why go through the effort of re-ripping or re-obtaining my music in higher quality later? The only real argument against using lossless music is that it takes too much space. With digital storage prices going the way they have, I see no reason not to invest in a relatively modest hard drive for music storage, it pales in comparison to the cost of other audio equipment!

With that said, my results showed that I was able to successfully differentiate all the way up to 320 kbps for two of the samples (Figures 2 and 7), up to 192 kbps for one of the samples (Figure 1), up to 128 kbps for one of the samples (Figure 3) and finally just to 96 kbps for three of the samples (Figures 4, 5 and 6). The ability to differentiate cuts off when I can no longer identify the difference to a statistically significant degree (as shown by the red bars in Figures 1-7 above).

Needless to say I was surprised by the variability I encountered in my abilities. While I was able to successfully differentiate all samples at 96 kbps, my inability to differentiate three samples at 128 kbps surprised me. Likewise, I was extremely surprised that I could differentiate between 320 kbps and lossless, I had previously thought people who said they could were talking out of their ass. The variability stemmed from the type of sound I was hearing, when the clip was quite melodic I had great difficulty picking out something to focus on. However, when some short, sharp sound was present (such as a snare drum), I could easily focus on it and differentiate all the way up to 320 kbps. The difference was essentially distortion or “fuzziness”, meaning that on the lossless version the sound was crisp and clear while on the MP3 it was blended together and lost the resolution.

This brings me to the following point. If you are an audiophile and invest even a meager amount into audio equipment, then what you are essentially doing from the hardware side is increasing the resolution of the music. That is, high end audio equipment will separate out the elements of the song and play them back clearly. If you then turn around and mask this by using lossy music, what was the point in buying the audio gear in the first place?

Also note, I used a modern encoder in this test (iTunes 8.2). Many people have music that is from dubious sources, encoded a long time ago with encoders that simply weren’t as good as todays are. When I listen to a 128 kbps song from that time, it reeks of distortion much more than the 128 kbps MP3s of today. This is not to mention the fact that some music obtained from dubious sources has been upsampled, i.e. ripped at bit rate X and then converted to bit rate Y which is higher than X. This results the appearance of higher quality via the bit rate number, but the song is of no higher quality than the original at bit rate X (think of shrinking an image to 100 x 100 pixels then blowing it up to 500 x 500, it will be all blurry).

Conclusion

In conclusion, my results indicate that I am able to differentiate lossy and lossless audio very poorly to very well, depending on the type of sample. Samples with short, sharp sounds are not encoded well so are much more easily differentiable than samples with smooth melodies.

Personally, I am very pleased with these results as I never thought I could actually differentiate that well. The results prompt me to continue my slow renewal of my music collection to lossless quality. On top of this, I now have confirmation that the feeling of better quality and depth I get when listening to lossless music is justified to some degree. It is often difficult to say whether the enjoyment we feel over something is simply due to the higher numbers we see (be they bit rates in music or dollar figures in an amp), and this is one case where I can say I have some empirical proof of my greater enjoyment of lossless music!

Appendix

This section contains some notes I made while doing the tests. It is most likely too boring for most people to read so I did not include it in the discussion. Notes are in order of testing.

Berlin_Drug: With lower bit rates, there was a high amount of distortion at the very beginning of the song, about 0-3 seconds in. At 192 kbps, it seemed easier to differentiate when X was the lossy sample. At 320 kbps it was hard, I mainly went on feeling rather than knowledge and seemed to choose B a lot as it was the last one I had heard when making comparisons.

King of the Mountain: Very hard! Even at 96 kbps. Could only tell difference due to a slight hiss in one sample compared to the other at 6.8 seconds. At 128 kbps, both samples had the hiss and I couldn’t find anything else to differentiate the two so gave up and skipped through the other bit rates quickly. May have found some differentiating factor had I lingered for longer, but it’s quite a boring task.

Welcome to Dryslexia: Obviously, the thing to focus on in this sample is the loud sound at around 2 seconds in. It is short, sharp and has a certain sonic sound to it that the lossy encoder simply can’t capture. Blazed through the samples, most of the time I didn’t even listen to A and B. I could just tell one was worse and knew B was the worse one so clicked X = B. Was very, very surprised I could differentiate easily between 320 kbps and lossless!

Nautilus: Very hard again. Could only tell at 96 kbps due to differences in the way the clip ends. After that no way I could tell.

Enchanting: Focus on the snare in the left channel 0.7 - 2.5 seconds into the song. This is easily differentiated at any bit rate by being “fuzzy” and distorted in the lossy sample and clear in the lossless one. Once again very surprised I could do this.

Finalfantasy: Focussed only on 0-0.1 seconds of the song, the very first thing you hear before the melody starts. Same as the others, distortion with lossy, clear with lossless.

Human_Disease: To be honest, this kind of music makes me cringe and this may have added to my failure to differentiate past 96 kbps. It seemed outright an easy sample due to the roughness of the sound but I just couldn’t take it, who would listen to this?!

General notes: Seems like a large part of ABX is finding that one thing that you can focus all your attention on and seeing how it varies between the two samples. A lot of the time you have to try many parts of the song to find this one thing. Without it, you end up relying on feeling which doesn’t get you much farther than 96 kbps. As for the relevance, does simply being able to tell that one piece of the song differs from the other enough to justify keeping lossless music? I think so, as when you listen to music it is composed of many parts and with some you may not tell, but with others you may be able to without knowing it. I would imagine it provides the music with a better overall feeling of crispness and sound separation, this is certainly what I feel when I listen to a lossless song compared to it’s lossy counterpart.

Like what you’ve read here?

Why not discuss it on the vel.co.nz IRC channel?

Click here!

Copyright 2009 by www.vel.co.nz