There is definitely diminishing returns, no doubt about it; adding a second stereo pair is far more important than going from the 7'th to the 8'th pair.
My point though is that we all have limited funds, and choose to spend that money where we feel best. I think the entire premise of this thread (that end users shouldn't be making sample sets) is quite flawed, and was just pointing out that there are multiple ways to get better sound out of our organs.
Suggesting that one can't be happy with the result of merging various samples is just as flawed as suggesting you can't be happy with just 2 or 4 speakers. My point is... why focus on just that particular aspect (user defined samples)? The OP might find that user sample sets are poor quality, and I might find that his audio system needs some work, but yet we don't go around saying "users shouldn't be building their own audio system by grabbing random speakers/parts of a shelf and assembling it because "I" might find flaws in the sound".