[BigMackCam]

I just came across this excellent article regarding sample variation between lenses. I've known and accepted, for some time, that sample variation exists - it makes sense when you realise that manufacturing tolerances have a finite range - but have always wondered why some reviews show a lens to be significantly better or worse than other reviews (also, why user opinions can vary so much). This article gives a really useful and accessible insight into the whys and hows of it, and it left me with a more realistic expectation of the reviews I read.

[mattt]

Quickly perused the article. They saved the best for last IMO: Speaking as a past quality professional, "As manufacturing variation goes to zero, production costs go to infinity"

[lytrytyr]

This topic has showed up on the site before.

See https://www.pentaxforums.com/forums/10-pentax-slr-lens-discussion/248765-advi...ml#post2654189 for example.

[normhead]

You might be interested in this series ....
LensRentals.com - Measuring Lens Variance

The simple fact that lens variation usually doesn't show up in prints is good thing. That people usually can't tell the difference between a print made with good lens or consumer kit lens, is also a bit telling. The MTF numbers that people stress about are often, undetectable differences, especially in prints. I recently returned a lens to Sigma complaining it was decentered, it was sent back with the AF system repaired and a couple of components replaced , but the did nothing for the decentering. They give you a 10 year warranty, but it only covers what they can fix.

[clawhammer]

The article at petapixel is written by Roger Cicala, the founder of LensRentals.com and the author of a lot of their blog posts on lens variations and all that can be learned by testing hundreds or thousands of lenses. He's probably the most knowledgable person about overall lens characteristics and sample variation outside of the manufacturers themselves (at least, among those people who write about this sort of stuff).

This article should be required reading for every forum member.

[savoche]

A good read.

I wish people could try to get this into their heads:

The laws of physics, manufacturing tolerances, and simple economics mean sample variation can never be entirely eliminated. Scream all you want about how you expect a perfect lens, but unless you’re willing to pony up $20,000 to $40,000 per zoom, you aren’t even going to get into the neighborhood of near-perfection.

[Alex645]

I love how you can go into a music store and play different guitars to help find "the right one", due to personal taste and sample variation. Imagine a camera store with new and used lenses that you could 'test drive' on your own camera with a lit studio and outdoor area with models. Back in the 80's after having 4 cameras and 6 lenses stolen, I had to replace all my Minolta gear and their HQ in Los Angeles humored me by opening an entire case of lenses to hand pick my glass. That lead to an awkward situation where I was displeased with the entire case (decentering, specks between elements, etc) which led me to Nikon and Pentax lenses (and bodies).

Because of sample variation, I've kept those prime jewels at the top of the curve and won't let go of them ever. On the other hand, when I've got the worst examples, I assumed it was just a junk design and never thought it could have been a reject that made it through QC. Of course, this is all pre-internet, so now with a minimal amount of research, we can better assess our expectations with any given lens.

Would we want the lens manufacturer to test each sample and rate them and charge accordingly so that a top scoring 40mm f/2.8 DA Ltd. was more expensive than an average score, and the low scores were highly discounted? Or do we like the uniformity of price with the chance of superiority or inferiority?

[Clarkey]

Quickly perused the article. They saved the best for last IMO: Speaking as a past quality professional, "As manufacturing variation goes to zero, production costs go to infinity"

To put it my favourite way, "the risk of getting something bad decreases based on the level of in-built quality".

In the real world however, there isn't such a thing as zero risk!

[MJSfoto1956]

Back in the day, Linhof used to get "first pick" of Schneider lenses -- these were branded differently than the remaining batch. Thus, people paid more for Linhof (and Leica too) lenses. I don't believe that happens anymore since the overall quality of lens manufacturing and design has increased dramatically in the past 15 years.

Michael

[normhead]

To me, the biggest take aways was, often, a good "bad lens" can be better than a bad better lens. When the difference between a measured

For example, looking at the measured MTF, (notice the article above says manufacturers cite the theoretical MTF based on the design) at it's best a DA 18-55 resolves 2604 @35mm ƒ8. The 18-135 resolves 2683 at 24mm ƒ5.6. The best a DA 16-85 does is 2612 at 24mm ƒ5.6. Given the degree of sample variation you could easily conclude, if you have those three lenses sample variation could change the pecking order of the resolution power of these lenses.

There will be times when the 18-55 will give you the highest resolution image. I'm not going to say "the best image" because for a zoom, the 16-85 has almost acceptable CA, the 18-55 WR has stellar performance CA control for a zoom. IN the centre any given lens could give you the best image, although not perceptibly so.

My advice remains, pick your lens based on the focal length and aperture. Newer lenses like the 16-85 are better edge to edge. Older glass have slightly better centre performance. There's no free lunch. For a traditional centre sharp, edge soft photo, the 18-55 is a great lens. Something that gets overlooked in the rush to upgrade.

[jkomp316]

common misconception about "making it through QC" with volumes as they are, QC is about tools to make good parts vs. checking parts after they're made. everyone got this misconception from the shiny gold inspector stickers manufactures used as a "feel good" measure.

[normhead]

common misconception about "making it through QC" with volumes as they are, QC is about tools to make good parts vs. checking parts after they're made. everyone got this misconception from the shiny gold inspector stickers manufactures used as a "feel good" measure.

No... you mean the was no Inspector #106? I'll sue.

[Nicolas06]

But what make a lens good or bad?

If for example I compare the DA15, DA21, DA35 ltd and FA77 on photozone at f/8. They have near the same performance. FA77 is a bit better on borders but the difference is negligible on numbers.

Now if I take actual pictures and look at them full screen, not trying to look at 100% crops: The FA77 and DA35 are always VERY sharp corners to corners. I but DA15 and DA21 have field curvature, meaning that depending on if the borders/corners are more near or more distant the focus point they will look sharp or soft. Not at 100% crop. No I have seen it in prints and just looking the full picture on screen.

And of course theses lenses have quite different max apperture, focal length, minimum focus distance, capacity to isolate subject from the background, bokeh, flare resistance and perspective distorsion characteristics.

When you speak of DA18-135, DA16-85, and DA18-55 this maybe not the MTF than make the difference:
- The range is different. Being able to shoot at 16, 85 or 135mm is important. You are not going to same results.
- The max apperture is quite different, the DA18-135 with its broad range keep a wider apperture sooner. That can mean less visible noise in the picture. The 18-55 on the opposite is quite limited with f/5.6 at 55mm and that's also limiting in my opinion for portraiture and in general isolating the subject.
- The 18-135 has weak borders in the 70-135mm range. This can be visible if you shoot a scene or landscape with it at such setting. This is not a problem vs 18-55 that don't cover this range at all, but versus 16-85, the story is a bit different.
- The prices of theses lenses vary greatly. The 18-55 is cheap but only bundled with a camera. The 16-85 is significantly more expensive.
- In term of size/weight the 16-85 is big compared to the 2 others.

But what maybe is the elephant in the room is that for the same price range you could buy a 17-50, 24-70 or 28-75 with all much larger max apperture for better low light performance and also more subject isolation (in particular for the last 2 at 70 ot 75mm).. Everybody will see the difference between a 75mm shoot at f/5.6 or f/2.8 even more so if there no so much light available. One would have maybe iso800, the other iso3200. 18-55 is really too short to cover most events and 16-85 as well as 18-135 have too narrow appertures for many cases.

[clackers]

But what make a lens good or bad?

Roger isn't talking about design performance, he's talking about copy by copy deviation from that performance.

He does have plenty of other essays on his website about design, though.

[Alex645]

But what make a lens good or bad?

If for example I compare the DA15, DA21, DA35 ltd and FA77 on photozone at f/8. They have near the same performance. FA77 is a bit better on borders but the difference is negligible on numbers.

Now if I take actual pictures and look at them full screen, not trying to look at 100% crops: The FA77 and DA35 are always VERY sharp corners to corners. I but DA15 and DA21 have field curvature, meaning that depending on if the borders/corners are more near or more distant the focus point they will look sharp or soft. Not at 100% crop. No I have seen it in prints and just looking the full picture on screen.

And of course theses lenses have quite different max apperture, focal length, minimum focus distance, capacity to isolate subject from the background, bokeh, flare resistance and perspective distorsion characteristics.

When you speak of DA18-135, DA16-85, and DA18-55 this maybe not the MTF than make the difference:
- The range is different. Being able to shoot at 16, 85 or 135mm is important. You are not going to same results.
- The max apperture is quite different, the DA18-135 with its broad range keep a wider apperture sooner. That can mean less visible noise in the picture. The 18-55 on the opposite is quite limited with f/5.6 at 55mm and that's also limiting in my opinion for portraiture and in general isolating the subject.
- The 18-135 has weak borders in the 70-135mm range. This can be visible if you shoot a scene or landscape with it at such setting. This is not a problem vs 18-55 that don't cover this range at all, but versus 16-85, the story is a bit different.
- The prices of theses lenses vary greatly. The 18-55 is cheap but only bundled with a camera. The 16-85 is significantly more expensive.
- In term of size/weight the 16-85 is big compared to the 2 others.

But what maybe is the elephant in the room is that for the same price range you could buy a 17-50, 24-70 or 28-75 with all much larger max apperture for better low light performance and also more subject isolation (in particular for the last 2 at 70 ot 75mm).. Everybody will see the difference between a 75mm shoot at f/5.6 or f/2.8 even more so if there no so much light available. One would have maybe iso800, the other iso3200. 18-55 is really too short to cover most events and 16-85 as well as 18-135 have too narrow appertures for many cases.

In this thread, the sample variation is purely a technical study in terms of resolution and probably CA. But we could have other discussions and ratings based on aesthetics, cost/value, depreciation, backward compatibility, size/weight, durability/weather or fungus* resistance or even other technical parameters such as AF speed or noise levels (in decibels not pixels).

It would be awesome if there was a super site that combined the best of DXOmark, Consumer Reports, and others that analyze and evaluate based on specific categories.

Inherently zooms, being more complicated, have a greater sample variation than primes. But I'd also love to see info on which manufacturers and even which specific lenses have more, or less, sample variation? Does anyone do any long term testing? Ultimately there probably isn't enough of a financial incentive to fund this sort of testing, other than manufacturers doing their own torture tests internally in the design and engineering phases.

And in the end, all those other parameters, both subjective and objective, is one of the reasons why this forum is so valuable.