[dexmus]

I was discussing K-5's high ISO capabilities with my friend and an interesting topic came up. If Pentax is indeed applying NR to raw files from ISO 3200 and above, and is getting such amazing results that are comparable to (or better than) FF cameras like D700 or 5DII, what is stopping the 'de-noise' softwares to improve and have the same NR algorithms, possibly even better?
If the de-noise softwares (Topaz, Noise ninja, even LR) start to implement the same NR logic what Pentax is using on the sensor - wouldn't it again give a 1-2 stop advantage to the FF raw images? Assuming ofcourse that the FF raw images have no artificial NR until ISO 6400.

Thanks

dex

P.S. - please don't comment on whether Pentax should or should not have applied NR. Not point of this discussion.

[Wheatfield]

We ran some very high ISO RAW files through both Topaz and Photoshop CS5 and decided that up to ISO 12800, the files were very usable. The guy who did the raw conversions is a Canon 7d user, and he felt the files were somewhat better than what his Canon gives up.
The stop above that (25600) was usable in a pinch, but should be avoided, and the top ISO was pretty much there for bragging rights, as nothing we did could tame the noise without turning detail into mush.
In point of fact, the Photoshop NR was somewhat better than Topaz, and also somewhat better than out of camera jpegs that had noise reduction done by the camera.

[falconeye]

I was discussing K-5's high ISO capabilities with my friend and an interesting topic came up. If Pentax is indeed applying NR to raw files from ISO 3200 and above, and is getting such amazing results that are comparable to (or better than) FF cameras like D700 or 5DII, what is stopping the 'de-noise' softwares to improve and have the same NR algorithms, possibly even better?

Little is known about the kind of NR Pentax does above ISO 1600, except that it isn't zero.

As a consequence, ISO where IQ drops to 30 dB SNR is on par with other vendors while ISO where IQ drops to only 24 dB SNR is ahead of other APSC vendors and approaching some FF results.

My personal assumption (pure speculation) is that the processing is very simple: Some sort of weighted binning of color channels like green and/or red/blue. Ie., a red Bayer sensel may be averaged with neighboring red sensels where the weight of the original sensel may be higher if it isn't too dark. If done this way (luminance-dependent weights) then it could be applied below ISO 1600 w/o DxO to take notice. Because only extremely dark tones then would be altered.

Such an operation makes sense!

If the difference between two neighboring sensels is more likely due to noise rather than a change in signal, then it would be better to combine results in order to make both probabilities equal.

However, this is normally best done in proper entropy-aware denoising software like Noise Ninja or Topaz Denoise. A simple algorithm like the one I just described can destroy an image.

But if done prior to demosaicing and with very conservative weights then it actually helps the demosaicing to return a more useful starting RGB pattern for ordinary denoising software to work with. Demosaicing makes quite some assumptions about a sensel's neighborhood and noise can destroy said assumptions destroying fine detail -- like when a sensel-wide line gets cut into points due to noise and points cannot be resolved by a Bayer sensor (their color would be unknown). However, embedded demosaicing-denoisers like DxO or LR3 would probably beat the Pentax approach.

All the fuzz could be avoided if testing labs would resample images to a standard 8MP prior to noise measurements (DxO does it, but after the measurement applying a mathematical model -- so, it doesn't help to eliminate this kind of effect).

[dexmus]

Thanks for the thorough explanation.


dex

[JohnBee]

Little is known about the kind of NR Pentax does above ISO 1600, except that it isn't zero.

Theoretically, if we took an ISO200 image and pushed processed it to ISO3200, we should bypass the effects of NR in contrast to a camera RAW shot at IS03200 correct?

[falconeye]

Theoretically, if we took an ISO200 image and pushed processed it to ISO3200, we should bypass the effects of NR in contrast to a camera RAW shot at IS03200 correct?

This is what most assume. Actually, to push ISO1600 to bypass raw NR as you wouldn't want to miss the analog gain which seems to increase until ISO 1600.

But this assumption implies that the raw NR is done independent of luminosities. DxO has not studied this. Their cross-correlation analysis wouldn't see alterations to very dark luminosities (because they are almost zeroes not adding to the overall correlation result). If I were Pentax, I'd apply raw NR below a given absolute luminosity threshold, not above a given ISO.

I looked into it a little bit at Falk Lumo: Lumolabs: Sensors of Nikon D700, D5000 and Pentax K-x: This compares the same D5000/K-x sensors and can be read as a study of Pentax NR.

You may look at the SNR curve or -10EV patches push processed. At ISO 100, K-x and D5000 are almost the same, except maybe below 0.1% (-10EV).

At ISO 1600, K-x and D5000 are identical again down to 1% and loose a stop against a D700. So, if Pentax uses NR at ISO 1600 below some luminosity threshold then it is below 1%. Which means that 18% gray measurements would stay accurate (free of NR) after pushing ISO 1600 to ISO 12800.

At native ISO 12800 (D5000 push processed from ISO 6400), the K-x gains 0.5 stops (compared to an ideal sensor) while D700 and D5000 loose ~1 stop. So, at ISO 12800, raw NR seems to gain Pentax 1.5 stops.

On the other hand, the ISO 12800 test patches (from the same article) still look best for the D700 and a quantitative study of noise at 50% resolution would reveal that numerically too.

[JohnBee]

Okay, there is most likely far more technical information that I will likely make use of(I'm more of a hands-on type of person). :P
However, based on what you've posted here and... on your current assessment of Pentax's sensor/NR technology. How would you interpret the following findings:

An ISO200 image exposed at ISO51200 yields identical grain and noise than a camera native ISO51200 image.

The same can be said for an ISO200 image exposed and shot at ISO1600/3200/6400/12800 etc etc... as each sample is push processed in RAW to proper exposure. Each consecutively showing identical grain, detail and noise levels as the camera native equivalents.

However... when it comes to the K20, K-x, K-7, and K-r, these ones show a dramatic shift in grain, detail and red channel information at a given point along the sensitivity curve.

I guess if I had one question to posit at this stage, it would be simply this.
Is it possible that the K-5 does not apply any NR based on sensitivities in contrast to previous models? And if not... how can observe or substantiate this phenomenon.

Please keep in mind when you answer that I'm approaching this from an end-user position. As I think many of us share also. And so if you could translate your answers into noob format, that would be very very good.

Thanks again for all your help with this btw.
I truly appreciate it.

[jpzk]

At the end of the day, we must then assume that there IS RAW NR being applied then?

With that in mind, at what ISO does it really kick in?

Not that I really care if Pentax apply NR at ISO 1600 and up, just that to me, any useable file at that high of an ISO is a bonus.

More to the heart of the subject of this thread ... I am lost with all of the very technical and scientific explanations.

JP

[falconeye]

More to the heart of the subject of this thread ... I am lost with all of the very technical and scientific explanations.

This thread ought to be very technical. The OP asked for it.

An ISO200 image exposed at ISO51200 yields identical grain and noise than a camera native ISO51200 image.

I don't think that this already is the case. I would assume that ISO 1600 has just a bit less read-out noise than 8x pushed ISO 200, i.e., that the analog preamplification adds slightly less noise than the ADC would do.

The common belief for the K-5 is that up to ISO 1600, analog preamplification is applied and above ISO 1600, raw NR. So, ISO 1600 is kind of a sweet spot for low light photography with the K-5 (combined with various amounts of negative EV compensation).

However... when it comes to the K20, K-x, K-7, and K-r, these ones show a dramatic shift in grain, detail and red channel information at a given point along the sensitivity curve.

AFAIK, K-x and K-r don't differ from the K-5 in this respect. But as I said, nobody really studied this yet.

K-7 and K20D have a noisy preamplifier and an off-chip ADC. Therefore, boosting ISO helps them a lot.

The lower the read-out noise, the less important is it to adjust an ISO level. The main source of noise (photon shot noise) is a natural phenomen and cannot be affected by adjusting the ISO level (it only depends on the total number of photons being detected by the sensor, i.e., lens aperture surface times exposure time times quantum efficiency of CMOS with the Bayer filter).

However, most vendors add black clipping to their higher ISO levels. If the clipping levels are smart (e.g., channel dependent), a higher ISO level may have less problems with tinted shadows than a pushed low iso image. But only because, e.g., LR only has a single black slider, not a black slider per color channel.

Is it possible that the K-5 does not apply any NR based on sensitivities in contrast to previous models?

Of course, but there is no evidence pointing towards this. Quick tests seem to put the K-5 on top of the D7000 for high iso images. So I guess it is the same NR. Moreover, Gordon B Good found the histogram be unequally populated above ISO 1600, but no empty bins. This is typical for bit shifting with NR being applied.

so if you could translate your answers into noob format

I'm sure if you ask a specific question, there will be enough fellow members to explain. I really have trouble to spend many words to say a thing which needs three words. Not only here on the forum


As for pushing ISO with the K-5, you may like the following thread:

La K-5 hace magia...

I showed the same effect for the K-x in my blog a year ago. But the pentaxeros story is much more easy going. For everybody lost in science

This is the Google translating link:
Google Translate

[JohnBee]

AFAIK, K-x and K-r don't differ from the K-5 in this respect. But as I said, nobody really studied this yet.

If you grab K-x RAW files from Imaging Resource and observe the red channel(red cloth) you will see chroma noise removal at work between ISO200 and ISO1600 samples.
Which has been typical of Pentax high ISO behavior for several generations. And I also think this is precisely what DXoMark refers too when they reference 'Smoothed' in their Pentax high ISO graphs.

ISO200 RAW


ISO1600 RAW


These samples were taken with NR_0 values.
Yet... the red channel(chroma) NR is very obvious as contrast detail is completely wiped by ISO1600.
Also you can observe the chroma noise removal present in the ISO200 files removed in the ISO1600 variant.
This behavior can be seen on the K20D, K-7, K-x and K-r.
I have not looked of examined RAW files from other models so I can't comment on them.

FTR. The K-5 on the other hand does not show such effects(red channel NR) at any point along the sensitivity curve(80-51200).



Let me know what you think of this and if you think it offers any real world significant toward the nature of the question.

[falconeye]

broken image links

[dexmus]

I would assume that ISO 1600 has just a bit less read-out noise than 8x pushed ISO 200, i.e., that the analog preamplification adds slightly less noise than the ADC would do.

The common belief for the K-5 is that up to ISO 1600, analog preamplification is applied and above ISO 1600, raw NR. So, ISO 1600 is kind of a sweet spot for low light photography with the K-5 (combined with various amounts of negative EV compensation).

Ok, so that means in Johnbee's experiment, when he pushes ISO 200 to 1600, he should see slightly more noise.

The lower the read-out noise, the less important is it to adjust an ISO level. The main source of noise (photon shot noise) is a natural phenomen and cannot be affected by adjusting the ISO level (it only depends on the total number of photons being detected by the sensor, i.e., lens aperture surface times exposure time times quantum efficiency of CMOS with the Bayer filter).

Of course, but there is no evidence pointing towards this. Quick tests seem to put the K-5 on top of the D7000 for high iso images. So I guess it is the same NR.

But then this conclusion directly contradicts johnbee's result -

The same can be said for an ISO200 image exposed and shot at ISO1600/3200/6400/12800 etc etc... as each sample is push processed in RAW to proper exposure. Each consecutively showing identical grain, detail and noise levels as the camera native equivalents.

What gives?

[JohnBee]

broken image links

Sorry, not clue what happened.
I didn't save them either(they were quick screen grabs).
However, the samples I'm working give a much bigger picture so I'll just post that instead I guess.

In short, what I was saying is that the NR in past systems is quite obvious.
While the K-5 shows no signs of this whatsoever(seems different.)

That's not so say noise filtering isn't applied, I'm sure there has to be something because the D7K has chroma at ISO200 whereas the K-5 is clean as a whistles(even pushed!). But... that's not what counts here either.

No one really cares how data is written from the sensor(per say). I think the hot issue here is that people are saying the K-5 is cooking RAW files at some sensitivity whereas others do not.

[AngryCorgi]

In short, what I was saying is that the NR in past systems is quite obvious. While the K-5 shows no signs of this whatsoever(seems different.)

I tend to agree with this. I looked at RAW files (DNG vs. NEF, both read in with RPP) of the K-5 and D7K and the shadow noise (intensity and pattern) for similarly exposed areas at ISO6400 and ISO12800 look absolutely identical. I wouldn't be able to tell them apart if my life depended on it. One of the following must be true:

(a) Pentax is NOT using secondary (primary being chip-level) NR on the RAW files...
(b) Pentax AND Nikon are using similar types of secondary NR on RAW files...
(c) Pentax is using the slightest of secondary NR on RAW files so that you can not discern it (option "c" makes absolutely no sense, but I am accounting for all possibilities logical or not)

I also don't see any detail loss in either that is indicative of secondary luminance NR.

That's not so say noise filtering isn't applied, I'm sure there has to be something because the D7K has chroma at ISO200 whereas the K-5 is clean as a whistles(even pushed!). But... that's not what counts here either.

Pure poppy-cock...I am looking at ye ol' Dino shot at ISO200 with no chroma NR applied at import and there is the same ever-so-slight chroma pattern in the midtones that the D7K shows. Pushed 1-1/2EV, and the chroma shows clearly in the shadows. This is the same behavior I'm seeing with the D7K (and pretty much every other camera I've ever owned...D700, D3100, D90, 30D, XT, XSi, 1D2, 5D...all show some chroma in shadows when pushing ISO200 files, all to varying degrees of course).

Not trying to burst any bubbles here...the K-5 images look fantastic, but I seriously can't say it looks any different than the other cameras with the 16MP Exmor in them.

[falconeye]

I tend to agree with this. I looked at RAW files (DNG vs. NEF, both read in with RPP) of the K-5 and D7K and the shadow noise (intensity and pattern) for similarly exposed areas at ISO6400 and ISO12800 look absolutely identical. I wouldn't be able to tell them apart if my life depended on it. One of the following must be true:

(a) Pentax is NOT using secondary (primary being chip-level) NR on the RAW files...
(b) Pentax AND Nikon are using similar types of secondary NR on RAW files...
(c) Pentax is using the slightest of secondary NR on RAW files so that you can not discern it (option "c" makes absolutely no sense, but I am accounting for all possibilities logical or not)

I also don't see any detail loss in either that is indicative of secondary luminance NR.

Please have a look at this:

(click and view in size "O")

The rightmost column, rows 2 and 3, compare K-x and D5k at ISO 12800 (D5k by boosting ISO 6400). You can see how the K-x has a "dampened" noise floor but text looks just a bit smoother too. This is the same situation but for Sony's 12MP chip. Moreover, DxO confirms that raw NR is still applied for the K-r.

Would you see a similiar difference in your K-5/D7k comparison?