Walked in on my first ever formal blind test...

[maxg]

On Saturday I had spent the day shopping with SWMBO and Junior and was quite at my wits end (you know what it is like - endless shoe and clothes shops and lots of stupid questions like - "does this make my hips look big?" and all that.

Anyway after a late lunch the 2 wanted to go off to a friends and I was set free. Normally shops close at 3 on a Saturday over here - but I passed by Stepcom just in case (favorite supplier).

Lo and behold - the place was packed. It seems that there has been an ongoing debate on some Greek audio site or other about differences in CD players. Some say there aint none - and that expensive ones are just rip-offs(?) - and some say there are differences and you get what you pay for.

Antonis, the owner of the shop, had taken such umbridge at the thought that all Cd players sound the same he had organised a blind test of 6 units for people to select the best.

By the time I had arrived that part had finished but there was a second part where they were about to compare the winner to 2 other models.

Some of the details on the test are as follows:

1. All CD players were hidden behind a curtain with the guy that was doing the connecting up. His communication with the audience was limited to announcing "CD Player number x" or taking requests from the audience to change units and/or music.

2. All players were connected to the pre-amp via some box from Behringer that allowed the output voltage to be fixed to a constant level from all players. Connections were all analogue. As the volume on the pre-amp was not touched during the testing this was assumed (correctly I think) to ensure no volume advantages for any unit.

3. Everyone, except me and one other who also arrived late, knew what units were being tested - but obviously not which ones corresponded to which number.

4. The methodology that I witnessed was to play a track on machine 1, then play a second track on the same machine. The player was then replaced by the second machine and the second track was played followed by the first again on the replacement machine. The audience was asked to choose one or other of the CD players, by a show of hands, to keep and that was then compared to a third machine using the same methodology above.

5. For reference the speakers in use were ATC and the amp and pre-amp were from a Norwegian company - Hegel. One of the CD players from the testing I took part in was from Hegel as well.

6. No machine was new out of the box. All bar one were owned by individuals who had brought them to be part of the test and the other one had been playing constantly in the shop for a week.

Some observations:

1. There was huge unanimity on selections. As voting was fairly instantaneous - and there was a minimum of interaction between audience members until the voting was complete it certainly appeared that everyone was largely in agreement of their own will and voilition.

2. Anyone who thinks that all CD players sound the same is barking - or has very limited hearing ability. The differences were startling - far bigger than I would have expected.

3. Although there was a clear winner there was general agreement after the even that this did not mean that the victorious machine would be better than all others in any given system. I have often beaten the drum of synergy on these forums and the general feeling of the audience was along those lines.

Results:

For the 3 machines that I witnessed (and took part in) the testing the rejection of the first machine was unanimous. The selection between the 2nd and 3rd machines was by a margin of 9 to 3 - with each side later recognising the relative strengths and weaknesses of each.

Analysis:

After all testing was done there was some discussion on what machines people thought they were listening to. It was impressive how wrong they all were as it turned out - with only one guy being spot on identifying the machine he owned.

It turned out that everyone was fooled by the most expensive machine - which came dead last. In actual fact we think there must have been a fault on it (I was not there for the testing of that unit). Even funnier - that machine was actually an SACD player - and would not select the CD track of the dual layer SACD in question so it should have had an unfair advantage over the others - but it worked against it. Of course - it could be that the SACD layer on that particular disk was very poor - even in comparison to the CD layer on the self same disk, but, the second track tested was from a normal CD and it did equallly badly on that one, apparently.

Machines in question:

Whislt I do not think the actual positions are really that important here - from what I can remember of the results:

1. Primare (normal CD player about 1500 euros retail).

2. Hegel CD player (2000 euros)

3. Marantz CD player using the Behringer as an external DAC (800 euros combined)

4. Harmon Kardon CD player (about 450 Euros retail)

5. Sony CD player (not sure on the model - looked like a 777 to me but it was certainly not one of their cheaper models)

6. Krell SACD player (4500 euros)

Surprises:

1. Obviously the Krell. Who knows what went wrong there? The owner looked suicidal at the end.

2. The Harmon Kardon. The cheapest player - but did very well considering.

3. The Hegel - everyone (except one) thought this was the winning machine and there were all sorts of discussions as to the advantages of its synergy with amp and pre-amp. There were some surprised faces I can tell you.

4. The Primare - I dont even know if this machine is sold in the states. It is sold quite hard over here - but with limited success and it is generally not that popular. The guy who owned it was cock-a-hoop of course - and others somewhat glum as their machines did badly.

Conclusions:

Differences - there were!

Pricing on equipment appears to be less important than synergy. Some items do seem to represent rather poor value for money. Buying anything without testing it first on your own system is perilous.

I genuinely had the impression this testing was done as fairly as possible and that the results were a true reflection of the quality of playback the audience perceived.

On another system the results might have looked very different but on this one the Primare was the clear winner. The Hegel, which was expected to win, appeared to be a little lethargic in its playback. Its bass was better - deeper, than that of the primare, but it lacked - dare I say it - PRAT (that will please Paul no doubt). There may be some minor power issue here but it definitely came across as slow.

3rd placed Marrantz/Behringer combo was as hard as nails. The violins screeched and were painful on the ear - but the overall impression despite that was actually rather good. On a less forward pair of speakers I think this one would do rather well.

The Harmon Kardon looks excellent value for money. It did way better than expected at half the cost of the nearest competitor here.

The Sony probably did about as expected. No-one seemed too surprised by its position.

[arfandbark]

What H/K CD player was that??????

My old Technics is tired................

[RAPTORMAN]

I wish your buddy Antonis, would have thrown in a nice TT on the blind test. I would really be interested on what the audience's comment about that.

[maxg]

I am told the HK CD player was the HD 970. May not be the same designation as used in the states - you need to check that.

As for throwing a pickup into the mix I guess the setup was too much of a nightmare to consider it - and it was beyond the scope of the test.

We actually discussed doing something similar with TT's in general but there comes a number of questions with that:

1. What arm?

2. What Cartridge?

3. What phono stage?

4. What platform/base?

+++++ about a dozen other factors. Also bear in mind that replaying someones audiophile vinyl again and again on the test would not be a good idea - and using multiple records risks introducing further variables.

[colterphoto1]

Balloting by show of hands would introduce a massive 'peer pressure' dynamic, skewing results. Private balloting would take longer but insure that no person was influenced to go with the 'group'.

Was the same selection/players tested more than once, in other words, the panel should have been tested numerous times, not in succession, to see if their own results were consistent.

Were these 'professional listeners' with golden ears or normal citizens

Again, for any statistics to be valid N (number of samples) must be greater than 30!

Sounds like a pretty fair test on initial observation, but most of these are completely full of holes and cannot be relied upon.

Michael

[maxg]

Balloting by show of hands would introduce a massive 'peer pressure' dynamic, skewing results. Private balloting would take longer but insure that no person was influenced to go with the 'group'.

2 things: 1. voting was pretty instantaneous and I dont think anyone felt pressured, and 2. Greeks - 2 Greeks - 3 opinions - they would vote against rather than with the crowd if pushed.

Was the same selection/players tested more than once, in other words, the panel should have been tested numerous times, not in succession, to see if their own results were consistent.

Had the results been closer we might have done that - but as it was it seemed very clear with the limited testing done. Also the guy that was behind the curtain was wiped out by the end - his was not an easy job.

Were these 'professional listeners' with golden ears or normal citizens

No professional listeners - but mainly audiophiles - or those that fancy themselves as audiophiles.

Again, for any statistics to be valid N (number of samples) must be greater than 30!

I think that applies to sampling where there is a large population. I am not sure how applicable it is to this event.

Sounds like a pretty fair test on initial observation, but most of these are completely full of holes and cannot be relied upon.

It was as fair as Antonis could make it - within the limits of his time and that of the listeners. Scientific - probably not - but good enough for all there.

[colterphoto1]

Didnt' mean to sound so critical Max, just that these 'tests' rarely have any real signifigance. But sounds like if hands shot up pretty quickly, various source material used (Diana Krall all night- geesh), and the guy 'behind the curtain' did a thorough job, that the casual audiophile might somewhat rely on the results, with the exception being the very bizarre results for the Krest unit.

Come to US next year and help out with the Pilgrimage testing, will ya?

Michael

[Colin]

very interesting, I have been to less formal events and the audience seems to jump quickly to a consciences on the sound of a particular piece of equipment, however many tweaking audiophiles will still prefer one complete system over another, though it seems as if that depends on whether they are focused on female voices (as am I) or hard rock[H]

[Erukian]

colterphoto1

I get what your saying. Peer pressure, your sitting in a crowd of self-proclaimed golden ears, so you want to hear the differences they do, so then there's a good change placebo will kick in altering your decision.

I still think that without these blind tests were completely lost. Next, i wanna see a interconnect test, then a speaker wire test, then a power cable test!

-Joe

[DrWho]

Well of course there was a difference....they were using the analog outputs! I would love to see the same test using the digital outputs into an external DAC. The only possible difference there would be jitter, but that usually isn't a problem with anything costing over $100 [] (heck, there are so few chips to choose from on the market anyway)

[D-MAN]

Max, great post!

DM

[maxg]

DMan - thanks.

Well of course there was a difference....they were using the analog outputs! I would love to see the same test using the digital outputs into an external DAC. The only possible difference there would be jitter, but that usually isn't a problem with anything costing over $100 [] (heck, there are so few chips to choose from on the market anyway)

We did talk about such a test - but the general feeling is that differences would be too small to notice as all the CD player would be doing is acting as a transport.

I hope, however, that you noted the volume leveling of the CD players in question. I thought this was a nice touch to the tests and apparently the variation in output between the units was quite considerable. Further, there had been some concerns over the Behringer masking the differences - but it appears that did not happen at all.

Oh - and FWIW - Antonis has sent the Krell off to the distributor with a note saying "there is something seriously wrong with this - FIX IT!"

We will see what they come back with.

[colterphoto1]

Yes Max, very good of you to emphasize the volume matching. As Who and I did with out informal Klipsch listening trials, it's very important to match volumes. As little as a couple of db can alter the sound somewhat and the louder source will almost always 'sound' better in a trial. So they definitely did the right thing there. Sounds like some level of care was taken, much more than many people generally get when auditioning components for purchase.

Michael