Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
06-22-2009, 06:55 PM   #46
Pentaxian




Join Date: Jun 2009
Location: GMT +10
Photos: Albums
Posts: 10,893
Original Poster
Mister Guy, their website at dxomark.com has plenty of info about their methodology.

Start from here: DxOMark Sensor

then look here:

DxOMark metrics
DxOMark Sensor metrics and photospace
Landscape photography: maximum Dynamic Range
Studio and portrait photography: Color Depth
Photojournalism & action photography: Low-Light ISO

then here:
Data normalization
and here:
Measurement definitions
and here:
Noise
and here:
Color measurements
and here:
Test Protocol
and here:
Print vs. original image

And then they document their testing protocols:
http://www.dxomark.com/index.php/eng/Technologies/Testing-protocols

including their testing facilties
http://www.dxomark.com/index.php/eng/Technologies/Testing-protocols/DxO-Labs...sting-facility
how they perform the noise measurement
http://www.dxomark.com/index.php/eng/Technologies/Testing-protocols/Noise-protocol
how they do ISO tests
http://www.dxomark.com/index.php/eng/Technologies/Testing-protocols/ISO-Sensitivity-protocol
and tests for color sensitivity:
http://www.dxomark.com/index.php/eng/Technologies/Testing-protocols/Color-Sensitivity-protocol

And they have 6 full pages of detailed discussion just on the subject of noise:
http://www.dxomark.com/index.php/eng/Technologies/Noise-characterization

etc etc etc.

I don't think you've been reading enough from their site.

PS - I don't work for them and have never even used any of their products. But I welcome their contribution to putting useful data out there on DSLR sensors.

In this regard I use dxomark in the same way I for example use the photozone.de tests for lens assessments.


Last edited by rawr; 06-22-2009 at 07:03 PM. Reason: Added even more links
06-23-2009, 05:14 AM   #47
Veteran Member
lol101's Avatar

Join Date: Jul 2007
Location: Garennes sur Eure France
Photos: Gallery
Posts: 900
QuoteOriginally posted by rawr Quote
Hi all.

What do you suppose the K7's high ISO performance will be like?

I'm curious since the high-ISO of the K20D and K200D (my current digital Pentax) is OK but not great, and this is proving to be a real obstacle to the kind of photography I like to do (street/ night/ nature).

I can work around current high ISO limitations with good camera craft, fast lenses, PP etc but quality high ISO performance gives valuable headroom.

From the DXOMark.com stats I've compiled a chart (see below) showing the relative performance of these Pentaxes visavis other APS-C sensors and current full-frame DSLR's. So far the Nikon D90 leads the APS-C pack and I must admit I am looking quite closely at it at the moment as a potential future upgrade..

I'd like the K7 to give the D90 a run for it's money in terms of high ISO, because I like everything else about the K7 and Pentax.

Does the K7 do high ISO significantly better than for eg the K20, or indeed up to the standard of the D90?
The problem is that these numbers are ponderated averages of averages and thus mean little by themselves.

Looking at the curves and using the "compare" feature in "print" mode (to somehow take into account resolution), you'll see that the K20's SNR is really equivalent to the one of the touted D300 and really close to the D90's (maybe 0,5 stop lower at most) so really, if we take as a starting point that the K-7 SNR characteristics won't be worse than the K20's, the performances will be as good as the D300 and very close to the D90.

If you want to get more than a stop worth of SNR, you have to go FF.

But there is one point where the K20D is markedly weaker than D90 (and 40D by the way) and that is DR at mid-ISO (200-1000) and that explains the higher "noise" score of the D90: it's not that the K20 has more noise, it's that it has less DR.
06-23-2009, 05:56 AM   #48
Pentaxian




Join Date: Jun 2009
Location: GMT +10
Photos: Albums
Posts: 10,893
Original Poster
QuoteOriginally posted by lol101 Quote
But there is one point where the K20D is markedly weaker than D90 (and 40D by the way) and that is DR at mid-ISO (200-1000) and that explains the higher "noise" score of the D90: it's not that the K20 has more noise, it's that it has less DR.
You are exactly right.

I think one of the issues many people have had in reading the low-light ISO scores on the chart, and the current Pentax APS-C sensor placements along the chart in particular , is that the low-light ISO number you see in the chart is not just about 'simple' noise.

If you go and look into how Dxomark craft that score, the ability of the camera to hold onto a certain dynamic range as ISO increases is a key part of the test - ie

Low-Light ISO is then the highest ISO setting for the camera such that the SNR reaches this 30dB value while keeping a good dynamic range of 9EVs and a color depth of 18bits.

So yeah, the D90/D5000's sensor is able to hang onto that 9EV of DR very well as you boost the ISO, and that characteristic must indeed have helped those two modern Nikon APS-C sensors score as well as they did.
06-23-2009, 12:59 PM   #49
Senior Member




Join Date: Jun 2008
Location: Atlanta, Georgia
Photos: Gallery
Posts: 182
The problem is that when they define Dynamic Range, they do that based on SNR (giving what one could call a "double dipping" in their score

Their own definition:
3. Dynamic range
Dynamic range is defined as the ratio between the highest and lowest gray luminance a sensor can capture. However, the lowest gray luminance makes sense only if it is not drowned by noise, thus this lower boundary is defined as the gray luminance for which the SNR is larger than 1. The dynamic range is a ratio of gray luminance; it has no defined unit per se, but it can be expressed in Ev, or f-stops.

Thus, if the camera is utilizing some noise reduction to improve the SNR, it will give a better DR score, even if it lead to some loss of "texture" in the image. Many do like the K20D images better (looking at examples from SpartanWarrior I prefer the K20D). And actually, their way of measuring DR will give as consequence that if you ran an image through e.g. Noise Ninja, the DR score as specified by DxO will improve.

I have to side with Dr. Falconeye (yes, he is a physicist, and I have found his posting based on very solid foundations and logic)
QuoteOriginally posted by falconeye:
They managed to produce a bag of numbers they don't properly understand themselves (e.g., it took them ages to start to realize that number of pixels does play a rôle ...). Let me say just this much: those DxO physicists wouldn't be able to publish their stuff in a renowed physical publication.
Best regards,
Haakan


Last edited by Haakan; 06-23-2009 at 01:36 PM. Reason: Corrected a misspelling
06-23-2009, 01:16 PM   #50
Senior Member
Mister Guy's Avatar

Join Date: Jan 2009
Location: Raleigh, NC
Posts: 244
QuoteOriginally posted by rawr Quote


etc etc etc.

I don't think you've been reading enough from their site.

PS - I don't work for them and have never even used any of their products. But I welcome their contribution to putting useful data out there on DSLR sensors.

In this regard I use dxomark in the same way I for example use the photozone.de tests for lens assessments.
I had read most of those, and just read a couple more. I think you're missing what I'm saying. They discuss how they measure things, and make some assumptions. That's fine, as long as they state them. Then they take those measurements, against test models, wave their hands, and predict how the camera will react in the real world. I'm not convinced the way they have chosen to isolate their tests is at all reasonable or valid.

They attempt to isolate noise with gray scale tests, presumably so they know what "right" is. They measure tonal range and dynamic range while ignoring noise, again, presumably so they know what "right" is. I'm not sure I agree with that. Since much noise is inherently electrical bleed and interference, I'm not sure a flat signal to noise ratio means anything, nor am I sure that the number of colors distinguishable reflects with any kind of accuracy the number of colors that ought to be present.

And I'm DEFINITELY not sure their score calculation is anything but gibberish. It's got 90% of one metric, 97% of the other, and 27% of the third, so clearly it's 89% as sensitive!

Er, eh?
06-23-2009, 02:24 PM   #51
Veteran Member
lol101's Avatar

Join Date: Jul 2007
Location: Garennes sur Eure France
Photos: Gallery
Posts: 900
QuoteOriginally posted by Haakan Quote
The problem is that when they define Dynamic Range, they do that based on SNR (giving what one could call a "double dipping" in their score

Their own definition:
3. Dynamic range
Dynamic range is defined as the ratio between the highest and lowest gray luminance a sensor can capture. However, the lowest gray luminance makes sense only if it is not drowned by noise, thus this lower boundary is defined as the gray luminance for which the SNR is larger than 1. The dynamic range is a ratio of gray luminance; it has no defined unit per se, but it can be expressed in Ev, or f-stops.

Thus, if the camera is utilizing some noise reduction to improve the SNR, it will give a better DR score, even if it lead to some loss of "texture" in the image. Many do like the K20D images better (looking at examples from SpartanWarrior I prefer the K20D). And actually, their way of measuring DR will give as consequence that if you ran an image through e.g. Noise Ninja, the DR score as specified by DxO will improve.

I have to side with Dr. Falconeye (yes, he is a physicist, and I have found his posting based on very solid foundations and logic)


Best regards,
Haakan
We are talking raw here and I don't think Canon or Nikon do anymore (or any less) "non-adjustable" NR on their raw files than Pentax.

The loss of texture can easily be seen on jpegs because other manufacturers have decided to do a rather large amount of NR at their default settings in jpegs while Pentax chose to do none (they actually changed their mind on the K-7 from what I gather) but when comparing raw files in the same converter and with 0 NR and sharpening, detail levels are really similar with a small advantage to the highest MP cameras.

Furthermore, their definition of DR looks coherent and it is clear that DR is limited (on the lower side) by noise: the noisier the shadows, the less DR in that region.
Noise and DR are related although not equivalent.
06-23-2009, 02:38 PM   #52
Veteran Member
falconeye's Avatar

Join Date: Jan 2008
Location: Munich, Alps, Germany
Photos: Gallery
Posts: 6,863
Looking beyond DxO

I have criticized DxO some posts above (not their numbers, but I questioned their ability to properly interpret them).

Because it is so much easier to criticize than to make a better proposal, I decided to share some bits of a discussion I had a year ago in a German forum, cf. DigitalFotoNetz.de :: Thema anzeigen - Fomag-Test: Eingangsdynamik und Rauschniveau where I sketched an alternative way of evaluating IQ.

For easier reference, I later fetched my contributions into a PDF (cf. attachment).

Please be aware that this is all in German language and for non speakers of the language, it really isn't worth to try to make sense out of all this I rather summarize it for you now.


1. State of the art, incl. DxO: noise is considered at the pixel level and measurbated in all possible ways. I.e., noise is only considered at one possible spatial frequency which isn't even typical or standard, but it is the highest possible, the Nyquist frequency.

2. Critics: The assumption that the cross-correlation function for pixel noise is a delta function is naive. On the good part, let's honor DxO for measuring the cross-correlation function for pixel noise. On the bad part, they would just assume that it is noise reduction applied to raw if they find a deviation. They never ever actually measure a signal ...
My other critics is that it is ridiculous (from a physicist's point of view) that a serious measure for IQ would depend on the existence of a digital sensor at all. What about IQ for film?

3. My proposition: Have a standard test chart with a fractal pattern where the signal covers all spatial frequencies. Then have the image mathematically transformed into spatial frequency space and use it to normalize a "constant or full signal".

Now, take photographs of the test pattern, transform, normalize and compare. Take many such images to determine noise. Don't just use the variation from one pixel to another. Of course, a laser projector (or lens clearly outresolving the spatial frequency integration range) must be used.

Then, you get a chart like the following (derived from mathematical considerations):

Blue: Signal curve plotted against spatial frequency (prototypical)
Red: Noise curve plotted against spatial frequency (prototypical)
Green: Quantization curve plotted against spatial frequency (prototypical)

The integrated surface between noise and signal can now be used to define a benchmark for a sensor's (or film's) image quality (at a given ISO step).

Please be aware that I totally ignored color here. But on the positive side, my measure would spot banding, clipping the blacks, or other serious and never measured artifacts.

[Source: all own science ]


When I do criticize DxO, then I mean it at a level they have defined themselves by the amout of effort they claim to put into. And then, I can only say they fail. They fail in a way that you simply cannot seriously say that the sensor with the higher DxOmark is the better sensor. And then, so what?

Last edited by falconeye; 06-15-2011 at 05:29 AM.
06-23-2009, 03:22 PM   #53
Veteran Member
filorp's Avatar

Join Date: Jun 2008
Location: Aberdeen Scotland
Photos: Gallery | Albums
Posts: 389
So... simplyfying the case, you would like to stop measure noise as it is - because it wont tell to us about quality of the sensor anyway, and instead you want to measure resolution at the different iso...., and on this ground to determine the real quality of the sensor.... what kind of calculation involve it?

06-23-2009, 03:48 PM   #54
Veteran Member
falconeye's Avatar

Join Date: Jan 2008
Location: Munich, Alps, Germany
Photos: Gallery
Posts: 6,863
QuoteOriginally posted by filorp Quote
So... simplyfying the case, you would like to stop measure noise as it is - because it wont tell to us about quality of the sensor anyway, and instead you want to measure resolution at the different iso...., and on this ground to determine the real quality of the sensor.... what kind of calculation involve it?
Yes and no.
I would like to measure the accuracy of reproduction of all (spatial) frequencies, very much like what is done in measuring high end audio. Normally, you have a drop-off of reproduction at very low and very high frequencies. Both are disturbing in images (i.e., banding, unequal exposure, pixel noise and flocking).

So, it is not about resolution as such. It is about reproduction of high resolving as well as low resolving patterns. The SNR of low resolving patterns can easily be in excess of 20 Bit! (its all in the German PDF ) But who did ever measure this?

Of course, max. resolution at various iso would automatically be involved because it is the point in spatial frequency where noise and signal curves meet (and therefore, intergration of my IQ mark would have to stop -- so normally, higher resolution at a given iso should mean better IQ in my definition, if noise at lower resolutions isn't too heavy).
06-23-2009, 09:03 PM   #55
Pentaxian




Join Date: Jun 2009
Location: GMT +10
Photos: Albums
Posts: 10,893
Original Poster
This conversation is heading in a direction that is exceeding my ability to comprehend it, alas.

The dxomark scores a useful metric imho. Not perfect, but useful and reflecting some degree of thought. That's enough for me.

falconeye's comparison of the issue with 'measuring high-end audio' is a signal to me about where this discussion seems to be heading - into the land of 'audiophiles' seriously discussing dubious matters like the pros and cons of US$10000 2m speaker cables.

Last edited by rawr; 06-23-2009 at 09:05 PM. Reason: typos
06-24-2009, 03:25 AM   #56
Veteran Member
falconeye's Avatar

Join Date: Jan 2008
Location: Munich, Alps, Germany
Photos: Gallery
Posts: 6,863
QuoteOriginally posted by rawr Quote
This conversation is heading in a direction that is exceeding my ability to comprehend it, alas.
You're right and I apologize to have captured the thread to this point. Will stop it here.
QuoteOriginally posted by rawr Quote
falconeye's comparison of the issue with 'measuring high-end audio' is a signal to me about where this discussion seems to be heading - into the land of 'audiophiles' seriously discussing dubious matters like the pros and cons of US$10000 2m speaker cables.
I meant the good old 70's when hifi was born. And minimum standards for "Frequency response", "Signal-to-noise ratio" and "Total harmonic distortion" have been set.
I am with you on your comment that many audiophiles have lost solid ground since.
06-24-2009, 09:10 AM   #57
Pentaxian




Join Date: Jun 2009
Location: GMT +10
Photos: Albums
Posts: 10,893
Original Poster
It is interesting to note from the PhotographyBLOG review of the K7 that the main reasons for the late-breaking changes to the K7 firmware/sensor are high-ISO performance related:

QuoteQuote:
Pentax's engineers have apparently "made a minor change to further improve the performance of the sensor on the camera to be included on the final retail version. As far as we are given to understand this will primarily reduce the amount of noise within the image; especially at higher ISO settings."
Obviously Pentax are very sensitive to the technical (and marketing) requirements around high-ISO performance nowadays.
06-24-2009, 11:54 AM   #58
Senior Member




Join Date: Jun 2008
Location: Atlanta, Georgia
Photos: Gallery
Posts: 182
QuoteOriginally posted by lol101:
We are talking raw here and I don't think Canon or Nikon do anymore (or any less) "non-adjustable" NR on their raw files than Pentax.
I will not claim I am an expert in this area, but there has been quite a lot of debate on DPR where people claim that they can show that the gain in the DxO score for Nikon D90 vs. D300 is only due to doing more aggressive NR at RAW in the D90. There has even been posts claiming to see typical NR artifacts in the D90 RAW.

IMHO I would rather have the RAW signal as "untreated" as possible, since I think that it is easier to upgrade external NR software, and getting more powerful processing in my computer, than getting these upgrades into my camera at a later stage, even if it takes its toll on the DxO score.

So if the allegations in DPR forum are true that the improved score for D90 is due to doing non-removable NR in RAW (other than the NR that is not possible to do outside the camera), I hope that Pentax does not go the same route for K-7, but rather keep the RAW as "raw" as possible.

(to be honest I am a little torn here since I realize that one one hand a higher DxO score might attract more people to buy Pentax, which is good. Then on the other hand, what if the higher DxO score comes at the expense of lower image quality, which is bad?)

Best regards,
Haakan

Last edited by Haakan; 06-24-2009 at 12:03 PM.
06-24-2009, 12:23 PM   #59
Senior Member




Join Date: Jun 2008
Location: Atlanta, Georgia
Photos: Gallery
Posts: 182
I was thrilled to find that my old K10D (introduced in 2006) has a better performance (DxO score) than the Mamiya medium format digital back introduced last year.

6000 dollars saved.
Attached Images
 
06-24-2009, 02:07 PM   #60
Senior Member




Join Date: Jun 2008
Location: Atlanta, Georgia
Photos: Gallery
Posts: 182
QuoteOriginally posted by falconeye:
I would like to measure the accuracy of reproduction of all (spatial) frequencies, very much like what is done in measuring high end audio. Normally, you have a drop-off of reproduction at very low and very high frequencies. Both are disturbing in images (i.e., banding, unequal exposure, pixel noise and flocking).
I do not understand German so I could not read the paper, but to my understanding, in simplified audio terms your proposal would be similar to having variable frequency generator or known white noise (=fractal pattern covering all frequencies) fed into the amplifier, and then measure the output signal and noise over all frequencies.

I think it makes sense. To me it seems as the way DxO measure the dynamic range is corresponding to measuring the signal on one frequency (homogenous grey over the sensor) and then measure noise at another frequency. In audio terms such a measure would be very easy to trick by just putting in a low pass filter. Of course the signal of the higher frequencies (= resolution), will also be affected, but that is not measured in the DxO measure.

Could a variant be to also use fractal pattern with two grey of different darkness instead of black and white. To my understanding in film, the resolution power of pure black and white patterns are different than for "grey-greyer" patterns, and there could perhaps be similar differences in digital. This would give an understanding of the possibility to properly capture more subtle textures.

Best regards,
Haakan
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
aps-c, camera, d90, dslr, iso, k7, k7 high iso, pentax, performance, photography
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
There is more to the K-5 than high ISO, right? jake14mw Pentax K-5 5 10-16-2010 09:33 AM
People High ISO K7 milesy Post Your Photos! 6 10-11-2010 01:36 AM
K7 at high ISO zelovoc Pentax DSLR Discussion 30 09-14-2010 07:23 AM
K-7 high ISO vs K20D high ISO supa007 Pentax DSLR Discussion 72 05-10-2010 04:24 PM
Night photography with K10D - High ISO short exposure VS Low ISO long exposure pw-pix Pentax DSLR Discussion 10 02-03-2008 01:37 AM



All times are GMT -7. The time now is 02:30 PM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top