Search:
Submit Search


The Great 16 Bit vs 24 Bit Challenge

A discussion forum for anything even marginally Hauptwerk-related.

The Great 16 Bit vs 24 Bit Challenge

Postby toplayer2 » Tue May 07, 2013 1:16 am

There is a strongly held belief amogst many Hauptwerkians that sample sets loaded in 24 bits sound superior to 16 bits. This experiment will give particpants the opportunity to test this for themselves in a rigorous and fair double blind trial. The need for a blind comparison arises from the profoundly powerful psychological tendency sometimes referred to as "expectation bias"; if we expect one thing to sound better than another, it will. Only if there is no possible foreknowledge of the objects under scrutiny is it possble to prove their audible differences. If 24 bit samples are superior, a critical listener should be able to reliably distinguish them from 16 bit samples. Reliability would be confirmed if the listener can correctly identify which is which in ten out of ten trials.

Ethan Winer led a panel discussion at the 2009 AES convention called "Audio Myths Workshop" that brilliantly details various types of bias that can affect our perceptions in powerful ways. This 58 minute YouTube video is highly recommended to anyone interested in this subject.

http://www.youtube.com/watch?v=BYTlN6wjcvQ

My friend and gentleman of impeccable integrity Steve Schlesing has very kindly prepared two short audio clips featuring the excellent 46 stop wet sampled Hereford VPO from Lavender Audio. The recordings were generated using a MIDI file prepared by Steve so that the performances are identical. Progressively more pipes were added to the registrations in each of the cadences. Reverb tails were allowed to decay. The only difference in the two versions is that in one the organ is loaded in 24 bits and in the other the organ is loaded in 16 bits. All other quality settings were set to maximum. The recordings were made in 32 bit floating point. Ten copies were made and saved as 24 bit WAV files. The file lengths were slightly altered in a random fashion. The files were given arbitrary names. Inspecting properties in a wave file editor will not provide any clues to differentiate the files. The links below are to these files in my Google Docs folder. There are somewhere between 1 and 10 files from the 16 bit source and the remainder from the 24 bit source. Only I know which named files are 16 or 24 bit. Any answers posted here will be given a score of 0 to 10 correct. Flipping a coin should allow for five correct guesses. Again, ten out of ten will be considered reliable. Good luck.

Note: the total size for all ten files is 139 megabytes.

https://docs.google.com/file/d/0B3LBjsV ... sp=sharing
https://docs.google.com/file/d/0B3LBjsV ... sp=sharing
https://docs.google.com/file/d/0B3LBjsV ... sp=sharing
https://docs.google.com/file/d/0B3LBjsV ... sp=sharing
https://docs.google.com/file/d/0B3LBjsV ... sp=sharing
https://docs.google.com/file/d/0B3LBjsV ... sp=sharing
https://docs.google.com/file/d/0B3LBjsV ... sp=sharing
https://docs.google.com/file/d/0B3LBjsV ... sp=sharing
https://docs.google.com/file/d/0B3LBjsV ... sp=sharing
https://docs.google.com/file/d/0B3LBjsV ... sp=sharing

From a technical perspective, the only difference other than the respective memory requirements between 16 bit samples and 24 bit samples is dynamic range. 16 bits provide 96 dB while 24 bits will in theory have a range of 144 dB, although in practice the limiting factor will be the user's audio chain. Pipe organs always have a certain amount of wind noise and therefore the dynamic range of a pipe organ is considerably less than 96 dB. Because of this, quantization noise will be inaudible due to masking. Although quantization noise is additive, so is the inherent noise in the samples, so it doesn't matter how many simultaneous pipe samples are rendered. The room you are sitting in right now is probably noisier than -96 dB SPL. What 24 bits does not do is provide a "higher definition" sound. It will not be clearer, more transparent, or changed in any way apart from the noise floor. Some will cling to the belief that 24 bit samples MUST be better. If anyone can discern 10 of 10 by ear in this experiment, then for that listener, 24 bits IS provably better. Other than using half again as much memory, there is certainly no harm.

Joe Hardy
User avatar
toplayer2
Member
 
Posts: 1051
Joined: Mon Jul 23, 2007 3:15 pm
Location: Michigan, USA

Re: The Great 16 Bit vs 24 Bit Challenge

Postby ggoode_sa » Tue May 07, 2013 5:18 am

Hi Joe,

I'm sending a private message to you with my answers :)

Just a few comments. I don't believe that the difference between 16-bit and 24-bit in audio reproduction is hugely noticeable (given that all other things are equal), but I do (for my own opinion) perceive a difference in higher harmonic clarity in the reverb when listening to things played back in 16 bit vs 24 bit (and freely admit that it could be my imagination). We'll see when you post back my score :shock:

What I do also notice is that different audio equipment makes the differences more or less noticeable. When playing the 10 demos back on the surround sound system that has ultra high clarity tweeters I perceive more of a difference than playing them back on the stereo system at our MIDI workstation. Playing them back through the Sennheiser HD 465s headphones is also different to the Sennheiser HD202 set... I could not play them back on my laptop as the audio card could only do 16-bit playback. It might all be expectation bias, but I also think that sending a 24-bit audio stream to the 24-bit DAC on my soundcard makes better use of the DAC. If I've got the higher resolution I may as well use it.

Kind regards,
GrahamG
ggoode_sa
Member
 
Posts: 363
Joined: Fri Nov 23, 2007 7:43 am
Location: South Africa, Johannesburg

Re: The Great 16 Bit vs 24 Bit Challenge

Postby toplayer2 » Tue May 07, 2013 7:02 am

Graham Goode lives up to his name. He scored a perfect ten. I have asked him to withhold his answers for now to give others a chance to give it a try.

Joe Hardy
User avatar
toplayer2
Member
 
Posts: 1051
Joined: Mon Jul 23, 2007 3:15 pm
Location: Michigan, USA

Re: The Great 16 Bit vs 24 Bit Challenge

Postby ReinerS » Tue May 07, 2013 7:06 am

Hi Joe,
I think this is a very interesting experiment! I'll carefully listen to these when I'm back at home this evening with my headphones (Sennheiser HD560 Ovation ii, 20 years old and still the greatest headphone I have ever owned) and send you my score as well. I'm very curious if I'll be able to discern these ;-)

For the benefit of everybody may I propose that we all refrain from posting our solutions but rather PM them to you so that we don't get an expectation bias from the solutions posted by others as well?

Best regards
Reiner
User avatar
ReinerS
Member
 
Posts: 766
Joined: Thu Oct 14, 2004 12:52 pm
Location: Braunschweig, Germany

Re: The Great 16 Bit vs 24 Bit Challenge

Postby adrianw » Tue May 07, 2013 7:43 am

Joe,

This is very humbling.

I have downloaded them all and listened on my iMac through good-quality headphones (Sennheiser 600s). TBH I really couldn't hear any difference at all.

Worse, I have just spent half an hour replaying them through my Cyrus amp + DAC. This gear cost me much more than I've admitted to my wife and has enjoyed good reviews acclaiming its transparency so ought to make it really easy to hear a difference. Nothing. Nada. I've flunked the test. I can maybe imagine a difference, and will certainly PM you some guesses, but my middle-aged ears seem not to be up to it.

I'm not sure whether to be worried about my hearing or pleased that getting a decent Hauptwerk set-up has just become a lot more affordable for me.

- Adrian.
adrianw
Member
 
Posts: 120
Joined: Sun Dec 18, 2011 6:10 am
Location: Lincolnshire, UK

Re: The Great 16 Bit vs 24 Bit Challenge

Postby sonar11 » Tue May 07, 2013 8:20 am

Adrian, It is humbling to be proven (mathematically) that you're wrong :) I include myself in that; a previous time this discussion came up I was a part of it and argued that I could hear a difference... turns out I was comparing an original sample vs a "extended" sample with the wind modelling turned off; the original was loaded in 24 bits, but I couldn't fit (memory wise) the extended in 24 bits due to the extra added ranks so I assumed that the bit difference was the reason, but it was not. Somebody (maybe Joe?) mentioned the whole "bits = dynamic range" fact on that thread and I did some digging around and discovered I was definitely wrong. So if you do hear a difference in samples, like "night and day" difference, then check and recheck how you're loading the samples, probably something else going on and not the bit-depth.

I did listen to the 10 test files, can't hear a difference either. However, I am pleased with that, because I've been loading all my samples in 16 bits to cut down loading speed and that means I haven't been missing out on anything :P (can never be sure when comparing samples loaded in 16 vs 24 because with the time it takes to recache the sample in a new bit depth, it totally resets your audio memory and at that point you are just making guesses as to which sounds better)
sonar11
Member
 
Posts: 647
Joined: Thu May 14, 2009 4:03 pm

Re: The Great 16 Bit vs 24 Bit Challenge

Postby etcetera » Tue May 07, 2013 8:39 am

PM sent.

I might be disqualified since I do spend waaaaay too much time listening to pipe recordings preparing for sample sets, so I might know what to listen for... Albeit my aging ears I did find some that were different from the others, but only when using phones and playing (too) loud. In the room with pretty good speakers, I really can't say.

And the question remains, is it better, or just different?


regards

Magne
etcetera
Member
 
Posts: 100
Joined: Thu Dec 03, 2009 5:56 pm

Re: The Great 16 Bit vs 24 Bit Challenge

Postby Eric Sagmuller » Tue May 07, 2013 12:19 pm

As I understand it bit depth 16, 20, 24bit has to do with dynamic range. In other words noise floor vs maximum volume. Sampling rate OTOH can have effect on the actual quality or clarity of details.

Eric
Eric Sagmuller
Member
 
Posts: 877
Joined: Fri Apr 11, 2008 5:18 pm
Location: Bellefonte, PA USA

Re: The Great 16 Bit vs 24 Bit Challenge

Postby toplayer2 » Tue May 07, 2013 1:38 pm

To Eric: you have it right on both counts. Sample rate is related to bandwidth. The approximate upper limit is ascribed by the "nyquist limit" or about half the sample rate. In theory 44.1 KHz would extend the audio bandwidth to 22 KHz. There are other quite audible artifacts that can occur if audio signal exceeds this limit and this is referred to as aliasing. This would be a good topic for another discussion.

To All:

Despite trying to be very careful to control conditions, sonar11 has brought a experimental design flaw to my attention that would allow a person using Audacity, Sound Forge, or the like to see a subtle difference in the displayed waveforms and thus be able to place the files into one or the other group. I am not certain that it would even be possible to eliminate this source of bias. So only if the participants choose to use their ears only will they be able to satisfy themselves about the outcome. Thanks to sonar11 for this information.

Joe Hardy
User avatar
toplayer2
Member
 
Posts: 1051
Joined: Mon Jul 23, 2007 3:15 pm
Location: Michigan, USA

Re: The Great 16 Bit vs 24 Bit Challenge

Postby John Murdoch » Tue May 07, 2013 1:54 pm

I'd love to participate in this challenge--but Google Docs is blocking downloads, saying the user's daily file download quota has been exceeded.
User avatar
John Murdoch
Member
 
Posts: 358
Joined: Fri Mar 18, 2011 9:06 pm
Location: Wind Gap, Pennsylvania

Re: The Great 16 Bit vs 24 Bit Challenge

Postby toplayer2 » Tue May 07, 2013 7:11 pm

John Murdoch wrote:I'd love to participate in this challenge--but Google Docs is blocking downloads, saying the user's daily file download quota has been exceeded.

Hi John,

Sorry for this inconvenience. I'll see if I can upgrade the daily limit. If not, presumably the quota will be reset tomorrow.

Joe
User avatar
toplayer2
Member
 
Posts: 1051
Joined: Mon Jul 23, 2007 3:15 pm
Location: Michigan, USA

Re: The Great 16 Bit vs 24 Bit Challenge

Postby ggoode_sa » Wed May 08, 2013 2:32 am

Hi again,

Thanks for additional info, and to Sonar11 for the report. Just to satisfy my curiosity I loaded them up in a spectrum analysis application and did an in-depth view of the differences. Once others have had a chance to listen and see if they can hear a difference, I'll post a couple of pics showing differences in the harmonic spectrum between the 16-bit and 24-bit versions.

Have fun!
GrahamG
ggoode_sa
Member
 
Posts: 363
Joined: Fri Nov 23, 2007 7:43 am
Location: South Africa, Johannesburg

Re: The Great 16 Bit vs 24 Bit Challenge

Postby toplayer2 » Wed May 08, 2013 6:06 am

In hindsight I should have asked Steve to make multiple recordings at each bit depth rather than just one of each. A blind test should be just that, there should be no possibility of the participant having any sort of foreknowledge. This morning I received a PM from a fellow forum member. He also was able to identify the test files as belonging to one of two groups by using Audacity to look at the digital fingerprints. Commendably, he reported that he and his wife had judged files to be either 16 bit or 24 bit based on listening alone before viewing the files. The answers were 4 out of 10 correct.

Joe Hardy
User avatar
toplayer2
Member
 
Posts: 1051
Joined: Mon Jul 23, 2007 3:15 pm
Location: Michigan, USA

Re: The Great 16 Bit vs 24 Bit Challenge

Postby John Murdoch » Wed May 08, 2013 10:46 am

First, thanks very much to Joe for putting this together--it's very instructive.

I've downloaded the files, and listened to them using the near-field monitors in my study at home. This setup is intended to be similar to my audio configuration at church--in practice, I think the home gear is more suited for accurate listening.

I figure if I can't tell a difference at home--nobody will be able to tell a difference at church. And (just to add a little bit of realism) there are thunderstorms threatening, so the dog keeps coming into the study to cower beneath my feet against the subwoofer. (I figure this approximates the overall audio impact of squirming two-year-olds and fussy newborns during morning worship.)

Joe--are these multiple instances of the same recordings? Or are these ten different records of the same MIDI playback, some recorded at 16-bit, the others at 24-bit? By that I mean, are we hearing the slight differences that we'd expect from a multiple-sample instrument?

Frankly--this is kind of encouraging. I use a notebook at home with a maximum of 8 GB of RAM--using a 20-bit or 24-bit depth would require replacing the notebook, which is a budget problem I'd like to avoid. This makes it plain that I can happily explore larger sound sets, using 16-bit depth.
User avatar
John Murdoch
Member
 
Posts: 358
Joined: Fri Mar 18, 2011 9:06 pm
Location: Wind Gap, Pennsylvania

Re: The Great 16 Bit vs 24 Bit Challenge

Postby OrganoPleno » Wed May 08, 2013 11:19 am

toplayer2 wrote:Commendably, he reported that he and his wife had judged files to be either 16 bit or 24 bit based on listening alone before viewing the files. The answers were 4 out of 10 correct.


Good Morning, Joe!

Thank you for posting this most interesting challenge. My wife and I accurately grouped the files into two groups based on wave-form in Audacity, but could not tell from that which group was which bit-depth. Fortunately, we had already recorded our SUBJECTIVE ASSESSMENTS of all the files, on which we were CORRECT an astounding 40% of the time.

Since this is now the time to be Subjective, let's revel in that a little bit. First, some background. I'm in my mid-60's, but recent audiology says "better hearing than expected for age", whatever that means. I do not own the Hereford sample set, so have no inside information about how that sample set is supposed to sound on one bit depth or another. I think I have a pretty good ear, developed by hand-tuning of my (home-built) acoustic Harpsichord. My wife has no musical training, but listens to lots of Organ Music with me... mostly of me trying to learn to play on Hauptwerk. She is an experienced observer with a PhD in Psychology, including studies of perception, both physiological and subjective (the nature of biases, and so forth). We tend to see things in a similar fashion. Remarkably, we were in 100% independent agreement as to our reported 40% accuracy in this test. Whatever it is we were hearing, we both heard it the same way.

The files were played back (under Windows 7 Pro) in Win-Amp, using an ASIO-driver plug-in, feeding to our RME Soundcard then through a PreSonus HP4 amp into our AKG K701 Headphones.

To avoid various biases as well as we could, we looked at the files for the order in which they were posted, and for the alphabetical order of their names (which was different), and for the order in which I downloaded them (which was neither of the above), and in order of increasing and decreasing file-size. Then scrambling the order of first listening so there could be no bias based on any of the foregoing.

For the very first file we listened to, at the very first trumpet blast, I said, "THIS IS 16-BIT SOUND!". Turns out I was correct. Subjectively, I thought the Trumpets were a little tinny. The second fanfare had a certain raspy quality, and the Full Organ section seemed a little thin and disappointing, not as rich as it might be. The whole recording seemed somehow a little distant or "filtered".

For the very next file we listened to, at the very first trumpet blast, I said, "THIS SOUNDS MUCH DIFFERENT, and much better... it must be 24-bit Sound". Again, it turns out I was correct. The initial trumpet blast sounded sweeter and clearer. The second fanfare sounded brilliant. The first section after the Trumpets sounded rich and mellow and satisfying, and the final section for Full Organ sounded very rich and deep. Clearly better in all regards. The whole recording gave a great sense of clarity and presence.

From there on, it was all downhill. Let's chalk it up to user fatigue and hearing the same thing too many times. Then we grouped the samples by our false estimates and played all of one together (finding none that stood out), then all of the other together (again, finding none that stood out).

Then we added our Dayton Audio "Puck" Tactile Sub-Woofers to "feel the bass". What an improvement to ALL the files! But on the file we first correctly identified as "24-bit", the deep Bass was exceptionally rich and smooth and seemed to roll beautifully from note to note, again giving a much greater sense of presence compared to the first file which we correctly identified as "16-bit". But nothing we heard let us change any of the false assignments we came up with, either.

Then we scrambled all the files into a different order and renamed them (by number) in this new arbitrary order (which I could not remember), then listened again in that order. It seemed much harder to tell anything apart this way, but the first two files which we correctly identified were identified the same way this time too, independently. Then hearing these files in this order sorted by their (false) assignments. Then "breaking the code" to compare second round with initial round judgements... Seven were same, three were different. Listening again to those three (without remembering which I thought was which), I ended up reverting all three to align with the initial judgements. (It turns out that one switched from wrong to right, and two switched from right to wrong.)

Then we tried using various meters and monitors during playback... which I had not used before and was not familiar with. It was possible to group the files into two categories, which exactly matched what I had seen in Audacity, but again could not tell which group matched which bit-rate.

It became evident that our subjective judgements did not align with the wave-form analysis. Assuming the wave-form to be definitive (as seemed most likely), our subjective tests were either 60% correct, or else 60% false. Turns out it was the latter.

And yet for the first two files we heard, the difference seemed so distinct. And, for those two files, our judgement never altered, regardless of playing order, and even when they were scrambled and re-named so we couldn't tell which ones they were. Interesting?

Watching a Spectral Analysis while listening, it gave the impression that no two files were the same... particularly during the "silences" (which are actually filled with complex background noise, possibly including the Organ Blower). Perhaps, given the complexity of the task, even an identical digital play-back comes out subtly different each time. Again, I could discover nothing systematic in this.

So that's the whole story. Two files, the first two we happened to listen to, clearly distinct from one another, and correctly (and consistently) identified as to bit depth (with clear preference for the 24-bit version). Then everything else sort of scrambled and unclear, sounding more and more alike the longer we listened.

For what it's worth.

Wishing a Good Day to All.

-- OrganoPleno
OrganoPleno
Member
 
Posts: 362
Joined: Thu Dec 03, 2009 5:08 pm

Next

Return to General discussion

Who is online

Users browsing this forum: No registered users and 3 guests