[beholder3]

Just a little warning for all beginner readers out there who do not want to invest too much time when thinking about which lens is good for them and then choose tabloid webpages like DxOmark with simplistic "scores".

While for non-enthustiasts it might sound a good lazy way to have a single score depict the technical (not visual) "quality" of a lens in fact it is not, never.

And it is never a good idea to assume that a commercial website which gives numbers over numbers actually know what they do better than plumber Joe.

One example how close to outright lying things can get with "scores" is the following:

Let's assume someone would want to compare the sharpness of both a Sony FE 90mm f2.8 Macro G OSS lens and a Canon EF 100mm f/2.8L Macro IS USM lens.

The good:
Sounds easy and straightforward, no? You "just measure it".

DxOmark lists the FE 90 as "42 P-MPix", while it has the Canon at "24 P-MPix", nearly half (!) for the Canon.
So doesn't this sound like the Canon is half as sharp as the Sony? Means: Much, much worse? Simple, isn't it?

The bad:
Ok, DxOmark fully ignores the fact that measuring a single lens item has pretty little meaning for average lens performance, so the values they present contain automatically quite some random deviation compared to what a photographer will see with his own lens copy.

The really ugly:
Let's look at what the industry standards of lens testing have to say: Test results from an optical bench.
Here you can have a look: Canon EF 100mm f/2.8L IS USM Macro Lens Average MTF

Whoa. The Sony is already worse than the Canon in the image center, but really falls behind big time when looking at the middle and edge performance. Tangential detail resolving performance for 10 and 20 lp/mm is like half of the Canon at the edges.

And yes, this is not only from the industry gold standard optical bench, but has been based on multiple lens copies as well.

So where Dxomark website claims a lens' sharpness is twice that of another product it actually is significantly worse. In other words you can not trust Dxomark unless you are happy with > 100% level of error in their "measurements" (which means you are happy with throwing dice for the results).


Presenting such misleading synthetic "scores" to a broad audience without big warning signs to lower skilled readers is as close to lying as it gets in my humble opinion.

Please note that "Complete scientific or technical documentation of the process used to compute P-MP values has never been published" according to Wikipedia. So Dxomark presents numbers they are not willing to explain even. They just make claims like any internet kid can do.
Not to be trusted. Not reliable.

[stevebrot]

Just a little warning for all beginner readers out there who do not want to invest too much time when thinking about which lens is good for them and then choose tabloid webpages like DxOmark with simplistic "scores".

The same is true of their sensor scores. The formula is to create a scoring algorithm based on dubious assumptions and described using unique jarthat will be difficult to debunk once firmly established in the target market.


Steve

[pschlute]

The way they score also means a lens tested on an aps-c sensor will score a lot lower than the same lens on a FF sensor !

[Rondec]

DXO Mark is really tough because a single number or a couple of numbers give you an impression of scientific accuracy, but what really tells the story are curves. At what aperture is peak sharpness? How are the edges as compared to the borders? At what apertures do the borders get sharp?

If we read DXO Mark's scoring, we should all be shooting with Sigma 85s on Nikon D850s, because that's what got top score on their charts.

I do think lens rentals has nice mtf charts that give not only mtfs, but also variance. Lens Rentals | Blog On that, Roger Cicala found a lot of copy to copy variance in the Sony 90 macro -- probably more than a high end lens like that should have, but as before mentioned, he is shooting the lenses using an optical bench, the gold standard and his tests are much more independent of camera body.

[CarlJF]

Let’s not forget that DxO warns users that lens scores can’t be compared between different cameras. The scores are for the specific lens+camera combo, not for the lens only. It’s not the fault of DxO if people don’t care about this and look at the scores blindly without taking time to minimally understand what they really mean.

That said, I’m not a big fan of DxO scores myself. Outside of bragging on the internet, they’re useless more often than not.

[cport]

Moreover, there are some other features which make sense in photography


Micro-Contrast, the biggest optical luxury of the world ? YANNICK KHONG

[Digitalis]

From my own testing with an optics bench I knew their results were far off the mark. In the early days I was hoping they would amend their rating system and scores - thus far they haven't, furthermore they have ignored calls to do so from people with far more expertise in optics and imaging than they claim to have. In academia it is ill advised to build your plans around a single source of data, use multiple sources if they exist, the truth will emerge from all the background noise.

[DWS1]

Just a little warning for all beginner readers out there who do not want to invest too much time when thinking about which lens is good for them and then choose tabloid webpages like DxOmark with simplistic "scores".

This is the root of the problem and why these type websites thrive. The purchase of a new lens is a major investment for most people, so take your time, determine your needs, research and gather information from multiple sources, rent or borrow a copy of what you think you want, then sleep on it for a day or two. If it then meets all your needs, fine, if not, start the search all over again.

So where Dxomark website claims a lens' sharpness is twice that of another product it actually is significantly worse. In other words you can not trust Dxomark unless you are happy with > 100% level of error in their "measurements" (which means you are happy with throwing dice for the results).

Isn't rolling the dice what we do every time we purchase a lens? I read here and elsewhere far too many times about people who are not satisfied with a new purchase and chalk their dissatisfaction up to ,"I must have gotten a bad copy." Is manufacturing variation and quality control really that bad to where it's a crap shoot to get a good copy?

BTW. I agree with you. Dxomark is misleading, biased, and has an agenda of using scientific type measurements to push people toward their sponsors products.

[Rondec]

This is the root of the problem and why these type websites thrive. The purchase of a new lens is a major investment for most people, so take your time, determine your needs, research and gather information from multiple sources, rent or borrow a copy of what you think you want, then sleep on it for a day or two. If it then meets all your needs, fine, if not, start the search all over again.



Isn't rolling the dice what we do every time we purchase a lens? I read here and elsewhere far too many times about people who are not satisfied with a new purchase and chalk their dissatisfaction up to ,"I must have gotten a bad copy." Is manufacturing variation and quality control really that bad to where it's a crap shoot to get a good copy?

I think the real thing to know is not that DXO Mark isn't totally worthless, but that there are better sources of data and reviews on the internet. We tend to quote their information, but as time goes by (a) it is less than complete (there is not testing of Pentax lenses on full frame cameras, their sensor testing has large gaps as well) and (b) it purports to allow for ranking of things that shouldn't be ranked.

Most testing sites will qualify that you shouldn't compare numbers across systems and with different sensors. But DXO Mark does exactly that and will give, say, a top score of '51' to a Sigma 85mm/D850 combo but that actually is meaningless to the average photographer.

[beholder3]

Let’s not forget that DxO warns users that lens scores can’t be compared between different cameras. The scores are for the specific lens+camera combo, not for the lens only. It’s not the fault of DxO if people don’t care about this and look at the scores blindly without taking time to minimally understand what they really mean.

Actually: no, they do not even meet this minimal standard.

Any user clicking on their "explore lens database" is not presented any warning of the kind you mention. If you see a warning there please let me know with a link. And obviously this is not a warning that would be sufficient to hide somewhere in the small print back pages of the site.

To the contrary: You are presented a list of all systems lenses with their "sharpness" score across all cameras and sensors. Basically the database itself does suggest it should be compared. It would cost the website coder 10 minutes to allow only showing lenses for one camera at a time - automatically filtered to only allow comparisons across what they say (?) is comparable.

Even worse they actively do compare lenses when advertising their new pieces of data. Click on any new lens advert on their site. Sample nonsense goes like this "In this review, we have compared the Nikkor Z 24-70mm f/2.8 S to its closest rivals. Remember that the lenses are intended to be used on different camera systems and mounts, so the comparisons are not strictly applicable."

Given the fact that a lens with actually superior sharpness can be given much lower scores as a result of their secret way of computing it the "not strictly" is a joke. Not at all is what it has to be. This type of paradox manipulation is what I feel is so close to lying.

It is like telling your 12 year old son to drink a bottle of beer and then a glass of wine but afterwards saying "but you should not strictly drink alcohol at your age".

[stevebrot]

Let’s not forget that DxO warns users that lens scores can’t be compared between different cameras. The scores are for the specific lens+camera combo

Yep...the most defensible use of their site is by the person assembling a new kit from scratch with little or no knowledge of what they are doing. Chose a body from list "A" based on sensor scores and a selection of lenses from list "B" based on P-MP. Easy-peasy!


Steve

[pentaxus]

Ah..now some things make sense. Back when I had decent test equipment and more time I tested many lenses. I found Pentax lenses to be quite good in general. When I paid attention to the DXO ratings Pentax lenses didn't do very well. They all had lower numbers because they were tested on APS-C cameras.

[BarryE]

Can't disagree with much of what's been said here. When I arrived in the digital photo world I was inclined to make decisions based slighty on what they measured, but not on the scores. Their measurement graphs, of say, DR do indicate differences in brands. For instance these graphs showed that the low ISO DR was better on Sony, Nikon and Pentax than Canon (the later using their own silicon, so no surprises). So a plus for them. Like any opinion, you are ill advised to just take one person's opinion, unless he/she is from this Forum ;-)

[6BQ5]

The only measurement from DxO that I like are the field maps. Their numbers give an indication of what you might possibly see. You would need to follow a couple different reviews and correlate the individual results. Assuming minimal variation between individual lenses of the same design, make, and model, I find the individual results generally track each other to some degree. Better indications can be found in sample images, from DPReview's Studio Comparison tool to real world images on sites like ePhotoZine. Some images can be used to determine corner sharpness while others can help determine contrast. If you can download the raw files and process them yourself then you will get more insight. Renting a lens is an even better test. Buying and using a specific copy under your own conditions is the ultimate best test.

[Larrymc]

Moreover, there are some other features which make sense in photography


Micro-Contrast, the biggest optical luxury of the world ? YANNICK KHONG

Very interesting article I can see his point in some of my lens!