While it seems appealing to be able to have the image quality of a camera represented by a simple number such as the DxO Mark, I am personally not convinced that we are there yet, especially looking at their low noise measure.
First, even while DxO produces a lot of measured values, it seems as there is quite a lot of subjectivity introduced when translating these values into composite numbers by the introduction of thresholds that has been set subjectively.
As one of the components in their low noise measures they use a threshold for the dynamic range, but IMHO their measure does not seem capture what is perceived as dynamic range when looking at a picture. While it would be nice, I do not think that an APS sensor camera (Nikon D90), at ISO 200, produces images on par with a 39 MPix Hasselblad MF format camera at ISO 50 in terms of how one perceive the dynamic ranges of an image. I think that a part of the issue is that the DxO score for dynamic range only look at the noise in the darkest area, and not if the darker area in itself is distinguishable to the next lesser darkest area. I.e. is it relevant to talk about an increase in dynamic range, if the extension of the range does not produce a distinguishable tonality? If taken to the extreme, a sensor which maps all signals below a certain threshold to pure black will have a very good SNR at the black areas, but can that be seen as a good measure to determine dynamic range?
For me, when looking at low ISO performance the image quality is much more related to the overall noise in the whole image, across all tonal ranges (this is the SNR score in DxO Mark), and to a lesser extent to the noise in the extreme darkest part of the image. Especially if the lower noise in the darkest area comes at the expense of not being able to separate out different shades in the deep darkness. I.e. can you claim a larger dynamic range if two nearby shades in the darkest areas are not possible to separate by tonal difference?
Since the DxO Marks for low noise performance of D90 compared to K20D and EOS50D has triggered quite a lot of discussions in different fora, I thought I would try to make my own comparison of the high ISO image quality using RAW files from Imaging Resource.
So I downloaded RAW files from Imaging Resource for K20D (K7 not yet available), D90 and EOS50D. I then converted them all in Raw Therapee, using "zero" settings for all values, except for exposure which was adjusted to give all images a similar white level. I used was the "multi targets" from Imaging Resource from which I cropped out the "Kodak Target" as a good representative image.
I have attached JPEGs of the resulting images below, scaled to the same sensor resolution (12 Mpix), 100% crops. When looking over the images, there seems to be some inconsistencies in the focusing, but since I only look at the noise structure, that should not be an issue.
When I looked at the resulting images, this is my take on them (note that this is subjective and others might come to other conclusions).
1. All 3 cameras are extremely capable and as indicated by the DxO SNR curves, they are all very close in overall "image noisiness", with a slight advantage to K20D at lowest ISO, in having 1 dB lower noise at its lowest ISO vs. the D90. I.e. at its best setting the K20D should produce a slightly cleaner overall images than the D90. But at higher ISO, there is a slight advantage to the D90.
2. Looking at the higher ISO images (ISO 1600 and 6400) for all tonal ranges, except for the darkest tones, there seems to be no big difference in the noise levels. To me they all give a similar "noise impression". The D90 looks cleaner in the darkest tones whereas the K20D looks cleaner in the lighter tones. Overall the EOS50D looks slightly noisier but the difference is quite small.
3. Looking at the character of the noise, I personally feel that the noise for K20D is somewhat smoother looking than for the D90. The D90 seems to have more "posterized" noise. When I cleaned up the images in "Neat Image" I felt I liked the result from K20D better than for D90. But the differences are really small and only found when pixel peeping. And I realize that since I have a Pentax, I might be somewhat biased in my finding. With the differences being quite subtle, I think that Nikon and Canon owners would be equally right in claiming a preference for their respective brands.
So, overall I feel that the DxO Mark, while a good attempt to try to help out in benchmarking performance, has not yet been able to fully correlate their numbers to how I perceive image quality, especially true for the composite numbers. And for some people it might be confusing when the presented numbers are a mix of logarithmic and linear numbers. E.g. color depths are given in bits (or EV), whereas the low light performance is given in ISO which is a linear scale. If converted to e.g. EV (and use a base ISO to put the number in a similar range as the other numbers), the low light ISO values for K20D, D90 and EOS50D could equally well be given as 19.3, 19.9 and 19.4 respectively (which might not be perceived as so dramatic differences as their linear representations). And these numbers are still very heavily influenced by the way DxO measures dynamic range, and not so much related to the overall all noise in the image.
So IMHO, despite the sometimes raging online wars on which camera of these three is the better, to me the differences looks quite minor at least in the above comparison, despite their difference in DxO scores
Now I have spent far too much time on this project, and will try to shoot some nice photos instead.
Best regards,
Haakan
K20D ISO 100
D90 ISO 200
K20D ISO 1600
D90 ISO 1600
EOS50D ISO 1600
K20D ISO 6400
D90 ISO 6400
EOS50D ISO 6400