Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
09-02-2009, 10:26 AM   #1
Senior Member




Join Date: Mar 2008
Location: Highland Park, IL
Posts: 211
When comparing bodies, lenses, brands: why not single- or double-blind tests?

I fairly regularly see in the online audio recording community the use of single- or double-blind tests to remove experimenter, placebo, and/or observer bias. (Obviously, assuming one has already ensured the test is performed as properly as possible.)

I've seen very few gear comparisons here on PF, or elsewhere in the broader online photography community, that utilize blind tests.

Just curious as to why? Any thoughts?

09-02-2009, 10:47 AM   #2
Ash
Community Manager
Loyal Site Supporter
Ash's Avatar

Join Date: Mar 2007
Location: Toowoomba, Queensland
Photos: Gallery | Albums
Posts: 22,692
Of course, double-blinded randomised controlled trials are the gold standard in comparison studies and testing hypotheses.

These work remarkably well in the medical and scientific world, for good reason, a little more difficult to apply in photography.

The investigator would be someone who knows a great deal about the gear and how it is used. You can't blind such a person from the gear being used. He/she can easily tell which body is the D300/700, which is the 5DMkII, and even by the process of elimination can identify that funny-looking obscure, unknown camera as a Pentax of some sort...

Lenses? Even less inconspicuous... Amateurs can tell which lens is which even without seeing the clear inscriptions on each lens! How much more a pro?

The only way this can be possible is if each camera and lens company produces their gear without identifiers (near-impossible) for a user to test and record.

That's one side. The second side is a little easier: what to do with the results - OK, so an independent reviewer can come and study each result without having to refer to the EXIF data. That's fine. So you'd have a single-blinded study making sure that the image identifiers are removed from the reviewer and nothing else that can possibly give away the camera/lens that created the image is shown.

This is likely the approach employed by the lens testing organisations, which would have to do as a double-blinded study is out of the question...
09-02-2009, 10:51 AM   #3
Ash
Community Manager
Loyal Site Supporter
Ash's Avatar

Join Date: Mar 2007
Location: Toowoomba, Queensland
Photos: Gallery | Albums
Posts: 22,692
Another thing that gets in the way of running trials - it takes time and money.
Who's going to fund such a study? as it doesn't benefit any particular company...
Pharmaceutical companies thrive on comparing their drug with a competitor's drug of the same class - I don't see lens companies doing this...
09-02-2009, 11:06 AM   #4
Veteran Member
ytterbium's Avatar

Join Date: Jan 2008
Posts: 1,076
You can make a camera "BLIND BOX" housing of some sort, where you have external access to generic controls. It would allow for a very limited review, not to show camera specific features (like viewfinders, LCD's).
Most likely nothing more than AF lock and shutter release, basic mode selection and one e-dial.

Best solution most likely is reviewing equal (RAW conversion, shooting conditions) printed images.
10 reviewers, each sorts images in ascending order quality wise, producing final scores.

What would confuse me more is lens selection. Do you choose equally priced or equal quality lenses for the same shots (to compare achievable quality or system price-performance ratio). If it is a camera review, why not use EXACT same (only one) lens for all systems - special inter-exchangable design (like adapt all), with outstanding quality - out resolving ALL cameras. If you look at the image-resource samples, i've seen the same tests preformed with 100mm lens on APS-C Pentax and 200mm lens on APS-C Canon. You get different DOF, perspective, rendering and diffraction limit. This all must be precisely defined and explicitly stated why it has been chosen to reach objective results.


Last edited by ytterbium; 09-02-2009 at 11:16 AM.
09-02-2009, 11:13 AM   #5
Senior Member




Join Date: Mar 2008
Location: Highland Park, IL
Posts: 211
Original Poster
Thanks for the comments so far. I should've left double-blind out of the original post, since that obviously poses significant challenges.

I still find it a bit surprising there aren't more single-blind tests and comparisons, both informal ones at PF and more formal elsewhere in the photog community. Not that I'm complaining! PF has proven an invaluable resource for me, I asked more out of curiousity than anything else.
09-02-2009, 11:24 AM   #6
Veteran Member
ytterbium's Avatar

Join Date: Jan 2008
Posts: 1,076
Maybe photographers are more artists than engineers or scientists that are used to do, read and interpret precision measurements. There are a lot more subjective reasons that can be of a big importance to photographer than such precision test.

The fact that there is a need for such tests to absolutely correctly determine best camera shows again, that the cameras have image quality so high and similar it is not the major factor differentiating them.
Or else it might be overly corrupt and biased review sites that position themselves as objective, neutral testers.
This suggests that maybe cameras should be evaluated by software, which allows you to define the goal for evaluation (maximum image quality, operation speed, high/low iso) and aid the user in such tests.
09-02-2009, 11:33 AM   #7
Senior Member




Join Date: Mar 2008
Location: Highland Park, IL
Posts: 211
Original Poster
QuoteOriginally posted by ytterbium Quote
Maybe photographers are more artists than engineers or scientists that are used to do, read and interpret precision measurements. There are a lot more subjective reasons that can be of a big importance to photographer than such precision test.
Interesting thought re artist v. engineer / scientist perspective. Single-blind testing need not be utilized strictly for precision testing and pixel-peeping, though. Single-blind testing can just as easily eliminate biases in subjective evaluations by the more artisticly-minded.

'Course, there's more to camera gear decisions than image evaluations, whether pixel-peeping or subjective. Big factors for me in picking my K10D: interface, overall usability, and it just felt "right" in my hands.
09-02-2009, 11:53 AM   #8
Ash
Community Manager
Loyal Site Supporter
Ash's Avatar

Join Date: Mar 2007
Location: Toowoomba, Queensland
Photos: Gallery | Albums
Posts: 22,692
But there are enough analytical thinkers in the photographic world... even on this forum.
The pixel peepers, mathematicians, IT and gear geeks as well as engineers - if there's a way to quantify image pop and contrast/resolution measures, they'll do it.

As for camera feel - that's very much subjective.

09-02-2009, 11:55 AM   #9
Veteran Member
nostatic's Avatar

Join Date: Feb 2008
Location: socal
Photos: Gallery
Posts: 1,576
I did something like this. I shot K20d, 5Dmk2 and D700 back to back to back in the same environment. I loaded the pictures into Aperture and showed them (2 and/or 3 up) to my g/f who didn't know which camera was used. She picked the one that looked "better" to her eye. Since she is an artist for a living, I kinda trust her eye.

The results were that she thought the D700 images were the sharpest but a bit cool. The K20d looked "best" but lost out when cropping tight. The 5D actually got unfairly compared as it was 100% crops which due to the extra resolution puts it at a disadvantage. When comparing similar size areas, the 5D and D700 were closer.

That's about as blind as I can do...
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
camera, community, dslr, photography, tests
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
Reasons to own camera bodies from different brands dragonfly Non-Pentax Cameras: Canon, Nikon, etc. 23 01-25-2010 04:17 PM
Comparing bokeh on 50's manual focus lenses. Voe Pentax SLR Lens Discussion 13 07-12-2008 06:13 AM
For Sale - Sold: FS: 2 Digital bodies, 2 film bodies, 4 nice lenses, 1 p-ttl flash kafreddy Sold Items 11 03-28-2008 02:36 PM
Comparing zoom lenses speed jjfdvm72 Pentax DSLR Discussion 12 03-11-2008 05:57 AM



All times are GMT -7. The time now is 09:34 AM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top