Forgot Password
Pentax Camera Forums Home

Show Printable Version Search this Thread
06-20-2009, 02:03 PM   #1
New Member

Join Date: Aug 2007
Location: Europe
Posts: 11
Some thoughts on lens reviews and methodology

Lens reviews in magazines and websites such as DPreview or Photozone seem to generate quite a lot of interest and discussions on the internet forums. But how good are tose reviews? In my opinion they are lacking severely in methodology and that is why I giving them less and less importance...

The problem that is most neglected in those reviews is reviewer bias. Reviewer bias does not mean that reviewers are intentionally dishonest. But measures should be taken that even unintentional bias is excluded. Usually it is done in the form of blind testing. Applied to lens reviews it means that the reviewers should not know what lens was used when evaluating image quality or technical test shots. They should purely focus on evaluating the images, without knowledge of the equipment used to make the images. A blind testing procedure could be as follows:

1) Technicians do the necessary measuring and they make al sorts of technical test shots. They should however not evaluate the measurements nor the test shots.

2) Different photographers (let’s say at least 3) will then take the lenses out for some intensive shooting. The photographers can comment on the practical aspects of the equipment tested (balance, robustness, etc.) but again, they should not evaluate the image quality of the pictures they made.

Of course it would be really difficult to do the first two steps blind.

3) Different experts (again let’s say at least three) should then independently evaluate the measurements, technical test shots and pictures made by the photographers. They would have to do that without having any knowledge at all about the lenses that are being tested. Perhaps the only thing that they are allowed to know, is some sort of lens category (ultra wide, wide, standard, short tele, extreme tele). They have to base their evaluations solely on the image quality itself. In short they should make an evaluation in absolute terms, without regard of brand name, price, aperture or whatever. Also, tests of different lenses should be mixed (and anonymised) so that all test shots and images will be evaluated in isolation (one picture is not necessarily related to another one). This would also avoid that for example an MTF resolution chart would have an impact on the evaluation of an image made with the same lens (if image sharpness is correlated with resolution it has to be observed independently).

4) The panel of technicians, photographers and image expert will come together to put the absolute image evaluations in perspective. Now brands, focal lenghts, prices etc. will be revealed to the image experts and the image evaluations can be put in perspective (eg. „low distortions for the focal length” or „very sharp for the price” ec.). Of course, the public should be given both the absolute and relative evaluations!

I don’t know if there are magazines or websites that use this kind of methodology. However I do think that such a review procedure bring a lot of surprises for all lens brands.


06-21-2009, 05:35 AM   #2

Join Date: May 2007
Location: WW community of Pentax users
Photos: Gallery | Albums
Posts: 2,384
Wouldn't this be a bit expensive? I count a lot of staff in your method.
06-21-2009, 05:48 AM   #3
Veteran Member

Join Date: Mar 2007
Photos: Gallery
Posts: 2,476
There cant be bias in evaluating measurements, if the resolution of a lens is measured to be a certain amount then it is that amount and you cant really judge that differently.
As for test shot evaluation, those are not in themselves such a good criteria for evaluating lenses because its hard to get the exact same shot from 2 lenses unless its a very controlled situation like a still life in a studio with artificial lighting.
and if you did do picture evaluations then some info would be required as the camera body enters into the equation. otherwise lenses could get better reviews because the reviewers like the output from the camera body it was used on more.
I think the biggest problem with lens reviews is sample variation in the lenses, that can greatly affect the rating of a lens.

  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
image, images, k-mount, lens, lenses, pentax lens, photographers, quality, reviews, shots, slr lens, test
Thread Tools Search this Thread
Search this Thread:

Advanced Search

Similar Threads
Thread Thread Starter Forum Replies Last Post
Suggestion Tamron lens reviews pacerr Site Suggestions and Help 49 09-27-2010 10:00 PM
Details on Lens Reviews legacyb4 Site Suggestions and Help 3 09-18-2009 03:29 PM
Lens reviews robbiec Pentax SLR Lens Discussion 28 07-08-2009 11:52 AM
Lens group replacement repair - DA* lens - thoughts? ksignorini Pentax SLR Lens Discussion 8 07-20-2008 05:53 PM
News The New Lens Reviews Section is here! Adam Site Suggestions and Help 29 02-07-2007 06:24 PM

All times are GMT -7. The time now is 04:56 PM. | See also:, part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]