[grhazelton]

One of the reasons I take reviews so lightly are the lack of double blind testing. There was a taste test done with beer many years ago, where the participants were all brand loyal drinkers who swore their beer was better than any other and that they could tell the difference. When given a taste of the 5 top selling brews they came it at exactly random choice levels. They couldn't tell the difference. The evidence is pretty much overwhelming, people can't make these kinds of decision accurately. Even a guy like Klaus over at PZ, can't make these kinds of decision accurately, even though he has the benefit of having 2D tests to help form his opinion.

If there's one thing you learn it's strength of conviction about accuracy does not mean one's opinion is more accurate. In fact those who have the strongest opinions are often those who have excluded the most relevant information in their research, focusing on one narrow aspect. Buy reducing a lens to an MTF number it's possible to ignore a lot of other relevant data, some of which are not even measurable. If psychology has taught us anything, it's that because something has some aspects that are quantifiable, doesn't mean the quantifiable aspect is the most important. It may be the most studied and tested, but that doesn't make it any more important. That just makes it the most talked about.

The one thing everyone can tell you, is if you buy the lens, can you get what you want to do done with it. Not maybe whether it's the best lens you can get for what you do, or if it will give you the most pleasing results based on your subjective analysis.

What most people want to know is if a certain lens matches their style. A reviewer probably can't tell you that.

Well put. I recently bought the Sigma 10-20mm f4. I'd have liked the constant aperture version, but for the price differential I can make "adjustments;" the variable aperture does just fine WITHIN ITS LIMITATIONS.

Sadly I'd imagine that few know what a double-blind test is. A prior post to this thread said that we see them often in this forum, I seriously doubt that! As I understand it a double-blind requires that neither the participants in the test nor the "operator" can know which images or whatever are being presented or in what order. Only the test "developer" knows which sample is which. All variables should be accounted for, in audio, for example, this would mean equalizing sound levels to something like .25 db.

Since we're unlikely to see such tests for lenses, it would be good if reviewers who furnish MTF and other hard data would suggest what variations from "perfection" are likely to be visable in actual use, and which can be safely ignored. Granted, this would be subjective, but the lens' output, a picture, is evaluated subjectively, isn't it? These "judgements" would aid the user in understanding the limitations of a lens - and ALL lenses have limitations! - and how to work around them. For example I find the 18 - 55 kit lens a good walk around user, certainly excellent in its price point.

[grhazelton]

One of the reasons I take reviews so lightly are the lack of double blind testing. There was a taste test done with beer many years ago, where the participants were all brand loyal drinkers who swore their beer was better than any other and that they could tell the difference. When given a taste of the 5 top selling brews they came it at exactly random choice levels. They couldn't tell the difference. The evidence is pretty much overwhelming, people can't make these kinds of decision accurately. Even a guy like Klaus over at PZ, can't make these kinds of decision accurately, even though he has the benefit of having 2D tests to help form his opinion.

If there's one thing you learn it's strength of conviction about accuracy does not mean one's opinion is more accurate. In fact those who have the strongest opinions are often those who have excluded the most relevant information in their research, focusing on one narrow aspect. Buy reducing a lens to an MTF number it's possible to ignore a lot of other relevant data, some of which are not even measurable. If psychology has taught us anything, it's that because something has some aspects that are quantifiable, doesn't mean the quantifiable aspect is the most important. It may be the most studied and tested, but that doesn't make it any more important. That just makes it the most talked about.

The one thing everyone can tell you, is if you buy the lens, can you get what you want to do done with it. Not maybe whether it's the best lens you can get for what you do, or if it will give you the most pleasing results based on your subjective analysis.

What most people want to know is if a certain lens matches their style. A reviewer probably can't tell you that.

Well put. I recently bought the Sigma 10-20mm f4. I'd have liked the constant aperture version, but for the price differential I can make "adjustments;" the variable aperture does just fine WITHIN ITS LIMITATIONS.

Wouldn't a proper double-blind test between the two Sigmas be interesting??

Sadly I'd imagine that few know what a double-blind test is. A prior post to this thread said that we see them often in this forum, I seriously doubt that! As I understand it a double-blind requires that neither the participants in the test nor the "operator" can know which images or whatever are being presented or in what order. Only the test "developer" knows which sample is which. All variables should be accounted for, in audio, for example, this would mean equalizing sound levels to something like .25 db.

Since we're unlikely to see such tests for lenses, it would be good if reviewers who furnish MTF and other hard data would suggest what variations from "perfection" are likely to be visable in actual use, and which can be safely ignored. Granted, this would be subjective, but the lens' output, a picture, is evaluated subjectively, isn't it? These "judgements" would aid the user in understanding the limitations of a lens - and ALL lenses have limitations! - and how to work around them. For example I find the 18 - 55 kit lens a good walk around user, certainly excellent in its price point.

[Kerrowdown]

Laboratory tests versus Real-life

IMHO lab tests are for folk that work in them as their day job.

For everyone else, lets get out there and make some stunning images and enjoy our photography.

[VisualDarkness]

Exactly, I love Klaus's numbers, I don't think his opinions are any more intelligent than anyone else's. I wouldn't buy a lens without checking Klaus' numbers. It's a great service he performs. His decisions on the "value" of a lens, well that's subjective, if the lens has what you want, it's value is amazing, if it doesn't have what you want, it's worthless. You can't say a lens is a good value, just because it's cheap. At least I can't, apparently others can. Ignore Klaus's star rating system and you'll be fine. His tests are bang on.

I think that as long as you don't mix up the pretty objective graphs and the subjective words I don't see the problem. Sometimes I agree with what he says and sometimes I don't, and by reading and thinking from my own view I can make my own conclusions of what the text means for me.

[dboeren]

Double blind certainly would be beneficial. For instance, recently I was studying the 3-way review on this site between the Pentax DA* 16-50 and the equivalent Tamron & Sigma lenses. There's a lot there about measuring sharpness, vignetting, etc... But what I'd really like to have seen is a process where they set up tripods and take the exact same pictures with the exact same settings using all three lenses and then pass them off (unlabeled) to a group of people to rank which ones they think look best. No machines, just pure visual appeal. I would find this very interesting, and might I add - I also find comparison reviews like this far more valuable than individual lens reviews because in the real world that's what I'm doing - trying to decide between several roughly equivalent options in a variety of price ranges. A similar shootout expanded to also include the Pentax 16-45 and 17-70 and incorporating a double blind would REALLY get my attention. That's the sort of thing that would get me to shell out ~$7 for a magazine issue containing such a test for instance.

[MSM]

The difficulty is that we look at testing for Objective information such as MFT graphs but in the end all that matters is the Subjective opinion of our output ie pictures. I agree the graphs can be helpful but the conclusions at PZ and others lets personal bias and subjectivity creep in. So I ignore that point. (It used to bother me that a lens from Pentax could get 3 stars and get a qualitative approval and yet the same scores of a Canon or Nikon would get a "thumbs up")

[KDAFA]

But its not just Klaus and PZ, is it?
Often in gear review magazines and websites you can see a clear difference in the language they use for Pentax gear (and other stigmatized brands, like Sony some time ago) and the discourse used for Canikon (and even more grandiose language for premium brands like Hasselblad and Leica). You can try this yourself, read a review of Pentax gear in some medium other than a Pentax forum and you will notice that there is a metaphorical "...too bad its a Pentax" somewhere in there. It will always be disguised, though. Sometimes this affects the item's rating, sometimes its just words in the text. Oh, and every so often, when the quality is undeniable, they will say "I can't believe its a Pentax!" in some way, thus implying that Pentax has no business making good products, that in fact Pentax products are or at least should be inferior.
To sum up, reviewers are very important when it comes to notifying potential customers about the value and worth of products. And reviewers are human and will often have some type of bias. And bias is rarely beneficial to outliers, it usually works for the mainstream, the popular; and punishes the others. Sony struggled a lot with this, but Sony has enough money for PR and advertising that they are slowly turning this around. Pentax doesn't appear to be doing that, at least outside Japan.

Yes, I know exactly what you mean about how Pentax seems to be viewed as somehow necessarily inferior. I've seen it happen a lot, in many places - and it's not good.

I think you've put it very well in your post - hit the nail right on the head.

[MSM]

I guess I just don't care any more about what someone else thinks - even a highly skilled technical reviewer whose opinions I read from time to time the way I read political blogs. Maybe I'm growing up or maybe I've run out of money or energy and I'm fooling myself. I certainly have grown tired of trying to defend my choices against the tide of popular opinion. The tide will rise and fall regardless of whether I want to.

As long as I avoid the true dogs, I likes what I owns and I owns what I likes. And I don't really care if Canon or Nikon or Sigma makes a "better" lens in a FL I already own since I don't own any of their cameras. Or whether a camera body that I like isn't very well received by the rest of the community, either.

I'd end with a, "So there!" but I like you guys and read your posts and they're interesting most of the time. But when I want or (think I) need something new I look here and at Flickr for examples of the product. That's all that matters to me.

LOL I am right there with you. I have really enjoyed our PF lens sample threads. That tells me more than most. I used to try Flickr but I find that I get a bunch of shots of beer cans, OOF shots, Asian women at auto shows, etc. Basically not what I shoot. In fact I have been turned off by some lenses after seeing the output of some lenses by the shots there.

Photography is subjective and trying to use only objective information to make a decision is going to come up short for me. There is too much bias (both intentional and unintentional) on the internet. So I choose to use more subjective approach. Do I like the images and does the lens fit my style of shooting?

One thing that has helped me here at PF is to look to the great photographers here. Folks like Jsherman and Simon are some that come to my mind. I am not at their level but seeing that I want to shoot similar things (family, kid shots) then when they post shots that I subjectively love, I have a bench mark as to the output potential of the lenses that were used. That is all I need to get started. The rest is up to me as far as whether I can afford the lens and if the weight, size, and focal length will work for me.

[NitroDC]

The only reassurance I need on my purchase is what I see after doing a shoot. My 15mm Ltd just does not stop amazing me in the details it captures. As well as my 55-300. The kit lens is another story but it is a KIT LENS.

I use the PhotoZone/DPReview/etc reviews only as a reference for sharpness, distortion, etc. The samples they take are always boring and unimpressive. I don't think I would have even considered the 15mm had I not stumbled upon the "15mm controls my mind" club. That thread alone made my decision for me.

You can't trust brick-wall and grid tests to make your purchase. Seeing what the lens can do in the hands of a (even moderately) talented photographer is KEY.

[MSM]

The only reassurance I need on my purchase is what I see after doing a shoot. My 15mm Ltd just does not stop amazing me in the details it captures. As well as my 55-300. The kit lens is another story but it is a KIT LENS.

I use the PhotoZone/DPReview/etc reviews only as a reference for sharpness, distortion, etc. The samples they take are always boring and unimpressive. I don't think I would have even considered the 15mm had I not stumbled upon the "15mm controls my mind" club. That thread alone made my decision for me.

You can't trust brick-wall and grid tests to make your purchase. Seeing what the lens can do in the hands of a (even moderately) talented photographer is KEY.

My point exactly! In fact IIRC the 15 mm lens was not a particularly popular lens for quite some time after it's release. But after the 'Mind control' thread (started by Jsherman), it saw many people flock to it like it was the greatest lens of all time. (It is quite good but too wide for my current needs). I often wonder what would happen if he had started a '21 mm Limited mind control' thread...

[joe.penn]

My 15mm Ltd just does not stop amazing me in the details it captures

Yep, exactly as published by PZ (the MFT charts):

Pentax SMC-DA 15mm f/4 AL ED Limited - Review / Test Report - Analysis

Of course it will amaze you in details, the MFT chart says so. MFT charts show resolving power which directly affects detail and sharpness. I can tell you without ever owning this lens or doing any research on it that it is extremely sharp wide open, and gets sharper stopped down a couple clicks - am I correct? Well, no need to really answer that because the MTF chart just told me the story.

A recent purchase of mine that I was a little leary on was the Tokina 11-16 2.8. I did a fair amount of research on this thing before purchasing it and was really stunned to see the MFT charts on this thing. I mean, come on PZ, your chart must be skewed on this one!

Tokina AF 11-16mm f/2.8 AT-X Pro DX (Canon) - Review / Lens Test Report - Analysis

A wide angle large constant aperture zoom lens from Tokina with massive resolving power at near every stop and every focal, it just can't be true! I had every intention on purchasing this thing just to try and sending it back and get another 8~16 - well, this thing is not going anywhere, the detail and sharpness from wide open and stopped down a bit is mind boggling.

As I noted before, PZ and Lenstip MFT tests are about as accurate as anything when gauging a lens optical performance, they are invaluable tools when researching lenses for purchase.

[lytrytyr]

Sadly I'd imagine that few know what a double-blind test is. A prior post to this thread said that we see them often in this forum, I seriously doubt that! As I understand it a double-blind requires that neither the participants in the test nor the "operator" can know which images or whatever are being presented or in what order. Only the test "developer" knows which sample is which.

Poster as "developer," Adam/site administration as "operator," forum respondents as "participants."
Your assignment may differ.

[grhazelton]

Yes, I know exactly what you mean about how Pentax seems to be viewed as somehow necessarily inferior. I've seen it happen a lot, in many places - and it's not good.

I think you've put it very well in your post - hit the nail right on the head.

Do have a look at Joe Farace's review of the K 30 in the current Shutterbug. Unfortunately it is not online yet. He is quite enthusiastic about the camera, likes the weather resistance and praises the shake reduction. He calls the autofocus "lightening fast," and says that J D Powers survey of initial quality calls Pentax number one. Check it out: 2012 Digital Single-Lens Reflex Camera Online Buyer Report | J.D. Power

Farace notes the pentaprism viewfinder, and mentions the unusual availability of interchangeable screens. All things considered I'd call it a rave review.

[KDAFA]

Do have a look at Joe Farace's review of the K 30 in the current Shutterbug. Unfortunately it is not online yet. He is quite enthusiastic about the camera, likes the weather resistance and praises the shake reduction. He calls the autofocus "lightening fast," and says that J D Powers survey of initial quality calls Pentax number one. Check it out: 2012 Digital Single-Lens Reflex Camera Online Buyer Report | J.D. Power

Farace notes the pentaprism viewfinder, and mentions the unusual availability of interchangeable screens. All things considered I'd call it a rave review.

In a day where Pentax is often sidelined, it's always a breath of fresh air to see things like these. The original post lists some other instances where Pentax receives high praise, and there are others still.

Clearly some do recognise the merits of Pentax, and say so.