[thirdofthree]

I fairly regularly see in the online audio recording community the use of single- or double-blind tests to remove experimenter, placebo, and/or observer bias. (Obviously, assuming one has already ensured the test is performed as properly as possible.)

I've seen very few gear comparisons here on PF, or elsewhere in the broader online photography community, that utilize blind tests.

Just curious as to why? Any thoughts?

[Ash]

Of course, double-blinded randomised controlled trials are the gold standard in comparison studies and testing hypotheses.

These work remarkably well in the medical and scientific world, for good reason, a little more difficult to apply in photography.

The investigator would be someone who knows a great deal about the gear and how it is used. You can't blind such a person from the gear being used. He/she can easily tell which body is the D300/700, which is the 5DMkII, and even by the process of elimination can identify that funny-looking obscure, unknown camera as a Pentax of some sort...

Lenses? Even less inconspicuous... Amateurs can tell which lens is which even without seeing the clear inscriptions on each lens! How much more a pro?

The only way this can be possible is if each camera and lens company produces their gear without identifiers (near-impossible) for a user to test and record.

That's one side. The second side is a little easier: what to do with the results - OK, so an independent reviewer can come and study each result without having to refer to the EXIF data. That's fine. So you'd have a single-blinded study making sure that the image identifiers are removed from the reviewer and nothing else that can possibly give away the camera/lens that created the image is shown.

This is likely the approach employed by the lens testing organisations, which would have to do as a double-blinded study is out of the question...

[Ash]

Another thing that gets in the way of running trials - it takes time and money.
Who's going to fund such a study? as it doesn't benefit any particular company...
Pharmaceutical companies thrive on comparing their drug with a competitor's drug of the same class - I don't see lens companies doing this...

[ytterbium]

You can make a camera "BLIND BOX" housing of some sort, where you have external access to generic controls. It would allow for a very limited review, not to show camera specific features (like viewfinders, LCD's).
Most likely nothing more than AF lock and shutter release, basic mode selection and one e-dial.

Best solution most likely is reviewing equal (RAW conversion, shooting conditions) printed images.
10 reviewers, each sorts images in ascending order quality wise, producing final scores.

What would confuse me more is lens selection. Do you choose equally priced or equal quality lenses for the same shots (to compare achievable quality or system price-performance ratio). If it is a camera review, why not use EXACT same (only one) lens for all systems - special inter-exchangable design (like adapt all), with outstanding quality - out resolving ALL cameras. If you look at the image-resource samples, i've seen the same tests preformed with 100mm lens on APS-C Pentax and 200mm lens on APS-C Canon. You get different DOF, perspective, rendering and diffraction limit. This all must be precisely defined and explicitly stated why it has been chosen to reach objective results.

---

Last edited by ytterbium; 09-02-2009 at 11:16 AM.

[thirdofthree]

Thanks for the comments so far. I should've left double-blind out of the original post, since that obviously poses significant challenges.

I still find it a bit surprising there aren't more single-blind tests and comparisons, both informal ones at PF and more formal elsewhere in the photog community. Not that I'm complaining! PF has proven an invaluable resource for me, I asked more out of curiousity than anything else.

[ytterbium]

Maybe photographers are more artists than engineers or scientists that are used to do, read and interpret precision measurements. There are a lot more subjective reasons that can be of a big importance to photographer than such precision test.

The fact that there is a need for such tests to absolutely correctly determine best camera shows again, that the cameras have image quality so high and similar it is not the major factor differentiating them.
Or else it might be overly corrupt and biased review sites that position themselves as objective, neutral testers.
This suggests that maybe cameras should be evaluated by software, which allows you to define the goal for evaluation (maximum image quality, operation speed, high/low iso) and aid the user in such tests.

[thirdofthree]

Maybe photographers are more artists than engineers or scientists that are used to do, read and interpret precision measurements. There are a lot more subjective reasons that can be of a big importance to photographer than such precision test.

Interesting thought re artist v. engineer / scientist perspective. Single-blind testing need not be utilized strictly for precision testing and pixel-peeping, though. Single-blind testing can just as easily eliminate biases in subjective evaluations by the more artisticly-minded.

'Course, there's more to camera gear decisions than image evaluations, whether pixel-peeping or subjective. Big factors for me in picking my K10D: interface, overall usability, and it just felt "right" in my hands.

[Ash]

But there are enough analytical thinkers in the photographic world... even on this forum.
The pixel peepers, mathematicians, IT and gear geeks as well as engineers - if there's a way to quantify image pop and contrast/resolution measures, they'll do it.

As for camera feel - that's very much subjective.

[nostatic]

I did something like this. I shot K20d, 5Dmk2 and D700 back to back to back in the same environment. I loaded the pictures into Aperture and showed them (2 and/or 3 up) to my g/f who didn't know which camera was used. She picked the one that looked "better" to her eye. Since she is an artist for a living, I kinda trust her eye.

The results were that she thought the D700 images were the sharpest but a bit cool. The K20d looked "best" but lost out when cropping tight. The 5D actually got unfairly compared as it was 100% crops which due to the extra resolution puts it at a disadvantage. When comparing similar size areas, the 5D and D700 were closer.

That's about as blind as I can do...