[BigMackCam]

Like any other site, DxOmark on its own has limited, potentially biased and questionable value... but depending on how you use it, I think it has some value when used as just one measure for a camera or lens. Let's face it, many folks are quick to agree with and reference DxOmark when its results support our own views and biases. We tend to criticise it more when they don't.

For my own part, I don't have a downer on DxOmark, but I'm as wary of its findings as I am with any other site or channel in isolation. The value of these various sources comes when their findings are combined with others to give a composite view. If three or four sites give generally positive reviews and data, and forum users are largely in agreement, we can be pretty confident that the reviewed item is decent, all things considered...

[beholder3]

Like any other site, DxOmark on its own has limited, potentially biased and questionable value... but depending on how you use it, I think it has some value when used as just one measure for a camera or lens.

Some replies:
a) anyone on the internet can provide valuable input, including Amazon buyer reviews. So yes, Dxomark plays in the same league as any single user on the net.
b) a site which advertises itself in the way dxomark does provides partial useful input mainly there where we all can follow how they come to their results. Very detailed and thorough documentation is required here. While measuring noise and dynamic range might be pretty straightforward (unless you go down the unreliable subjectively twisted route that Bill Claff does on his homepage) presenting "scores" without publishing the exact way how they compute them is worst practise. Documentation is easy to do. Not presenting it is always done for a (bad) reason.

One of the reasons why I do like lensrentals reports is that they are very good at putting disclaimers in and stating what you can NOT take from their findings.

What we are discussing here is pretty much the question if one would like their child to come back from school with a math exam graded "F" and no reasons, corrections, detailed errors given whatsoever. "It's a score the teacher has chosen".

All any commercial website needs to do is document how they get their results so readers might be able to reproduce them with effort.

[Trickortreat]

Just a little warning for all beginner readers out there who do not want to invest too much time when thinking about which lens is good for them and then choose tabloid webpages like DxOmark with simplistic "scores".

While for non-enthustiasts it might sound a good lazy way to have a single score depict the technical (not visual) "quality" of a lens in fact it is not, never.

And it is never a good idea to assume that a commercial website which gives numbers over numbers actually know what they do better than plumber Joe.

One example how close to outright lying things can get with "scores" is the following:

Let's assume someone would want to compare the sharpness of both a Sony FE 90mm f2.8 Macro G OSS lens and a Canon EF 100mm f/2.8L Macro IS USM lens.

The good:
Sounds easy and straightforward, no? You "just measure it".

DxOmark lists the FE 90 as "42 P-MPix", while it has the Canon at "24 P-MPix", nearly half (!) for the Canon.
So doesn't this sound like the Canon is half as sharp as the Sony? Means: Much, much worse? Simple, isn't it?

The bad:
Ok, DxOmark fully ignores the fact that measuring a single lens item has pretty little meaning for average lens performance, so the values they present contain automatically quite some random deviation compared to what a photographer will see with his own lens copy.

The really ugly:
Let's look at what the industry standards of lens testing have to say: Test results from an optical bench.
Here you can have a look: Canon EF 100mm f/2.8L IS USM Macro Lens Average MTF

Whoa. The Sony is already worse than the Canon in the image center, but really falls behind big time when looking at the middle and edge performance. Tangential detail resolving performance for 10 and 20 lp/mm is like half of the Canon at the edges.

And yes, this is not only from the industry gold standard optical bench, but has been based on multiple lens copies as well.

So where Dxomark website claims a lens' sharpness is twice that of another product it actually is significantly worse. In other words you can not trust Dxomark unless you are happy with > 100% level of error in their "measurements" (which means you are happy with throwing dice for the results).


Presenting such misleading synthetic "scores" to a broad audience without big warning signs to lower skilled readers is as close to lying as it gets in my humble opinion.

Please note that "Complete scientific or technical documentation of the process used to compute P-MP values has never been published" according to Wikipedia. So Dxomark presents numbers they are not willing to explain even. They just make claims like any internet kid can do.
Not to be trusted. Not reliable.

Thing is - DxOmark measures performance of the camera+lens combo so it cannot measure the absolute performance of a lens tested like optical benches do.

[beholder3]

Thing is - DxOmark measures performance of the camera+lens combo so it cannot measure the absolute performance of a lens tested like optical benches do.

Thing is - they actively nudge unskilled readers to compare across camera+lens combos and make the same mistake themselves. No excuses possible there.

And as it stands it is a unfounded claim that they actually measure anything any more than any John Doe on the internet does when he says "great lens".

Any child can claim a non-documented score is what they "measured". Evidence? Zero.


On the "liar detection" score that is a plain 10 out of 10 for dxomark.

[Trickortreat]

Thing is - they actively nudge unskilled readers to compare across camera+lens combos and make the same mistake themselves. No excuses possible there.

And as it stands it is a unfounded claim that they actually measure anything any more than any John Doe on the internet does when he says "great lens".

Any child can claim a non-documented score is what they "measured". Evidence? Zero.


On the "liar detection" score that is a plain 10 out of 10 for dxomark.

Dunno, their results compare favorably with other imatest based benchmarks. Optical benchmarks are all good and fine, but they don't tell the whole story to users because no user will use just the lens itself. Lens on camera tests seem to show more of what can you get in real world use.

[Desperado]

As time goes by in photography those internet pages comes very ridiculous,Lightening,composition,and model posing would become your musts.Even pentax 18-55 kit lens does great job.

[Trickortreat]

Yeh. I have found that also true. Even the crappiest lens can produce wonderful results. But benchmarks go so well with marketing. How would a manufacturer sell you that new $2000+ lens if there were no benchmarks

[Rondec]

Like any other site, DxOmark on its own has limited, potentially biased and questionable value... but depending on how you use it, I think it has some value when used as just one measure for a camera or lens. Let's face it, many folks are quick to agree with and reference DxOmark when its results support our own views and biases. We tend to criticise it more when they don't.

For my own part, I don't have a downer on DxOmark, but I'm as wary of its findings as I am with any other site or channel in isolation. The value of these various sources comes when their findings are combined with others to give a composite view. If three or four sites give generally positive reviews and data, and forum users are largely in agreement, we can be pretty confident that the reviewed item is decent, all things considered...

I have found that even when it comes to sensor scores, they significantly overstate dynamic range. My guess is that somehow they measure there being shadow detail when there is actually mostly noise. Regardless, the site Photons to Photos has charts that seem more real world. So, for the K-1, DXO Mark measures it at 14.6 EVs of dynamic range at base iso and Photons to Photos measures it at 11.3. To me, the second number is a lot closer to what I usually get at iso 100 on a K-1.

I suppose the value is overstated similarly for all cameras so it doesn't really matter, but it does give a semblance of accuracy that maybe isn't quite there.

[BigMackCam]

I have found that even when it comes to sensor scores, they significantly overstate dynamic range. My guess is that somehow they measure there being shadow detail when there is actually mostly noise. Regardless, the site Photons to Photos has charts that seem more real world. So, for the K-1, DXO Mark measures it at 14.6 EVs of dynamic range at base iso and Photons to Photos measures it at 11.3. To me, the second number is a lot closer to what I usually get at iso 100 on a K-1.

I suppose the value is overstated similarly for all cameras so it doesn't really matter, but it does give a semblance of accuracy that maybe isn't quite there.

I haven't used Photons to Photos before. Sounds useful... I'll take a look. Thanks!

[Trickortreat]

I have found that even when it comes to sensor scores, they significantly overstate dynamic range. My guess is that somehow they measure there being shadow detail when there is actually mostly noise. Regardless, the site Photons to Photos has charts that seem more real world. So, for the K-1, DXO Mark measures it at 14.6 EVs of dynamic range at base iso and Photons to Photos measures it at 11.3. To me, the second number is a lot closer to what I usually get at iso 100 on a K-1.

I suppose the value is overstated similarly for all cameras so it doesn't really matter, but it does give a semblance of accuracy that maybe isn't quite there.

Those 2 sites just take different S/N thresholds as base where they start to measure.

[beholder3]

I have found that even when it comes to sensor scores, they significantly overstate dynamic range. My guess is that somehow they measure there being shadow detail when there is actually mostly noise. Regardless, the site Photons to Photos has charts that seem more real world. So, for the K-1, DXO Mark measures it at 14.6 EVs of dynamic range at base iso and Photons to Photos measures it at 11.3. To me, the second number is a lot closer to what I usually get at iso 100 on a K-1.

I suppose the value is overstated similarly for all cameras so it doesn't really matter, but it does give a semblance of accuracy that maybe isn't quite there.

I guess nobody cares about what number there is actually. Photons to Photos is just using undisclosed and shady mechanisms to compute their "scores" just as dxomark does with lenses.

It is not documented in a way to be reproducible and their choice of ignoring a range of existing data is absolutely arbitrary. In this case dxomark's actually is not, they use all data.

The problem with Claff is he decides for all readers what shall be important data to them.

[Trickortreat]

Its a black box. If the data is compatable readers shouldnt worry whats happening behind the screens. They couould just as well have a base camera wizh score 100 and all other cameras compared with a percentage of base camera

[BigMackCam]

Its a black box. If the data is compatable readers shouldnt worry whats happening behind the screens. They couould just as well have a base camera wizh score 100 and all other cameras compared with a percentage of base camera

Actually, this is where I see the value of sites like DxOMark, Photons to Photos, etc. Not in terms of absolute values, but the performance of each camera relative to others... i.e. does camera X have better or worse dynamic range than camera Y at the ISOs I'm interested in? If the general consensus from multiple sources says camera X has better dynamic range, and if that should matter to me, then it's a big tick in the box for that camera.

Am I over-simplifying this, or missing something, perhaps?

[Trickortreat]

Actually, this is where I see the value of sites like DxOMark, Photons to Photos, etc. Not in terms of absolute values, but the performance of each camera relative to others... i.e. does camera X have better or worse dynamic range than camera Y at the ISOs I'm interested in? If the general consensus from multiple sources says camera X has better dynamic range, and if that should matter to me, then it's a big tick in the box for that camera.

Am I over-simplifying this, or missing something, perhaps?

Not really. Photographers shouldnt break their heads what happens behind the scenes. My view on matter is same as yours

[Digitalis]

where I see the value of sites like DxOMark, Photons to Photos, etc. Not in terms of absolute values, but the performance of each camera relative to others...
...If the general consensus from multiple sources says camera X has better dynamic range, and if that (result) should matter to me, then it's a big tick in the box for that camera.

Fundamentally, this statistical qualitative analysis approach is sound. After all, every test group needs a statistical outlier. A data point so far off the plot it causes the analysis team to scratch their heads, and posit the question: "what went wrong there?" Which in the case of DxOmark, is a question in desperate need of a candid response.