[bsbxl]

Lens reviews in magazines and websites such as DPreview or Photozone seem to generate quite a lot of interest and discussions on the internet forums. But how good are tose reviews? In my opinion they are lacking severely in methodology and that is why I giving them less and less importance...

The problem that is most neglected in those reviews is reviewer bias. Reviewer bias does not mean that reviewers are intentionally dishonest. But measures should be taken that even unintentional bias is excluded. Usually it is done in the form of blind testing. Applied to lens reviews it means that the reviewers should not know what lens was used when evaluating image quality or technical test shots. They should purely focus on evaluating the images, without knowledge of the equipment used to make the images. A blind testing procedure could be as follows:

1) Technicians do the necessary measuring and they make al sorts of technical test shots. They should however not evaluate the measurements nor the test shots.

2) Different photographers (let’s say at least 3) will then take the lenses out for some intensive shooting. The photographers can comment on the practical aspects of the equipment tested (balance, robustness, etc.) but again, they should not evaluate the image quality of the pictures they made.

Of course it would be really difficult to do the first two steps blind.

3) Different experts (again let’s say at least three) should then independently evaluate the measurements, technical test shots and pictures made by the photographers. They would have to do that without having any knowledge at all about the lenses that are being tested. Perhaps the only thing that they are allowed to know, is some sort of lens category (ultra wide, wide, standard, short tele, extreme tele). They have to base their evaluations solely on the image quality itself. In short they should make an evaluation in absolute terms, without regard of brand name, price, aperture or whatever. Also, tests of different lenses should be mixed (and anonymised) so that all test shots and images will be evaluated in isolation (one picture is not necessarily related to another one). This would also avoid that for example an MTF resolution chart would have an impact on the evaluation of an image made with the same lens (if image sharpness is correlated with resolution it has to be observed independently).

4) The panel of technicians, photographers and image expert will come together to put the absolute image evaluations in perspective. Now brands, focal lenghts, prices etc. will be revealed to the image experts and the image evaluations can be put in perspective (eg. „low distortions for the focal length” or „very sharp for the price” ec.). Of course, the public should be given both the absolute and relative evaluations!

I don’t know if there are magazines or websites that use this kind of methodology. However I do think that such a review procedure bring a lot of surprises for all lens brands.

Bsbxl

[Bart]

Wouldn't this be a bit expensive? I count a lot of staff in your method.

[and]

There cant be bias in evaluating measurements, if the resolution of a lens is measured to be a certain amount then it is that amount and you cant really judge that differently.
As for test shot evaluation, those are not in themselves such a good criteria for evaluating lenses because its hard to get the exact same shot from 2 lenses unless its a very controlled situation like a still life in a studio with artificial lighting.
and if you did do picture evaluations then some info would be required as the camera body enters into the equation. otherwise lenses could get better reviews because the reviewers like the output from the camera body it was used on more.
I think the biggest problem with lens reviews is sample variation in the lenses, that can greatly affect the rating of a lens.