Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version 29 Likes Search this Thread
05-03-2023, 11:52 AM   #1
Loyal Site Supporter
Loyal Site Supporter




Join Date: Jul 2021
Location: SF Bay Area
Posts: 45
FYI: Applying AI noise reduction to K-3 III raw files

The following are my observations.

AI-based noise reduction is a very powerful tool to eliminate or reduce noise.
Adobe has just launched its own AI Denoise, but unfortunately, it does not work with K-3 III DNGs (no NR applied).
DxO's PureRAW and PhotoLab 6 contain two relevant options. Unfortunately, the newer DeepPRIMEXD seems to generate artifacts, while DeepPRIME works well.

I hope this is helpful information.

05-03-2023, 12:14 PM - 1 Like   #2
Loyal Site Supporter
Loyal Site Supporter
UncleVanya's Avatar

Join Date: Jul 2014
Photos: Gallery | Albums
Posts: 28,401
Does this happen with KP, KF, K70, or K1ii files?

---------- Post added 05-03-23 at 03:22 PM ----------

I wonder if this could be affecting more than the Pentax gear: https://photographylife.com/reviews/dxo-deepprime-xd. Reading this suggests that artifacts are getting created more routinely than we might wish.

Last edited by UncleVanya; 05-03-2023 at 12:22 PM.
05-03-2023, 12:39 PM - 2 Likes   #3
Pentaxian
simon_199's Avatar

Join Date: Oct 2014
Posts: 613
I have already dug into this topic, and even opened a dedicated thread on DxO support forums. Other users weighed in, but so far I've seen no improvement in DeepPrime XD for KIII mk3, which is unfortunate. Lately I have downloaded a few samples from the usual DPreview studio set, from various cameras and my conclusions are as presented in the thread (where perhaps I mentioned no raw files being available in DPreview database for K3iii, but they are available now). For almost any camera DPXD is essentially black magic. Fuji 26mpix and even the new 40mpix sensors in particular are incredible (XD support for Xtrans was added in a recent update). I have tried full frame big mpix cameras, micro 4/3 cameras...always XD yields great results out of the box, better than DP. With the K-3iii even standard deepprime has "subpar" performance, but XD is totally broken. It always degrades image quality by introducing artifacts and actually increasing noise in out of focus/uniform areas. Completely opposite behavior with respect to other cameras.

This is an indirect result of Pentax's questionable choice of applying a level of in-camera noise reduction baked into the raw that is significantly stronger than any other camera on the market. Pentax cameras with accelerator chips are an outlier in the training dataset. The statistical properties of the noise are unlike any other camera. Therefore to adapt the model to this specific type of file would require a level of fine tuning that is probably not considered worth the effort by DxO. As a matter of fact, DP and XD work worse on the K1ii than on the K1, see for yourself on DPr samples. I have to say that the KP behaves pretty well in spite of the accelerator. So there is hope that they could fix it sooner or later.

Luckily the problem is worse at very high ISO, where it is unlikely that image quality is critical. Up to ISO4000-5000 deep prime (non XD) does a decent job on the K-3iii. Yes it works much better on other cameras with (essentially) the same sensor like the Fujis. But up to that sensitivity the difference is at the level of "pixel peeping". At 12800 ISO performance is really poor in comparison to other cameras.

Possible problem in DeepPrime XD support for Pentax K-3iii - DxO PhotoLab - DxO Forums

Advanced AI denoise like DxO (I am not a fan of Topaz), fully integrated in a powerful raw processor, is really a game changer. For most cameras (unfortunately not the K-3iii...) it is set and forget, just leave DPXD always enabled at default settings and it does the trick. Artifacts are very rare and occur if you push it too much (which is not needed). Almost every camera, including M4/3 sensors, becomes usable up to ISO12800, or more. The amount of detail that it is able to salvage is staggering. Among the samples I have tried I particularly like the output of "high density" sensors like the new fuji 40MP, clean yet natural; and the advantage with respect to "standard" denoise algorithms is clear as day for those sensors.

Last edited by simon_199; 05-03-2023 at 12:45 PM.
05-03-2023, 12:53 PM   #4
Loyal Site Supporter
Loyal Site Supporter




Join Date: Jul 2021
Location: SF Bay Area
Posts: 45
Original Poster
QuoteOriginally posted by UncleVanya Quote
Does this happen with KP, KF, K70, or K1ii files?

---------- Post added 05-03-23 at 03:22 PM ----------

I wonder if this could be affecting more than the Pentax gear: https://photographylife.com/reviews/dxo-deepprime-xd. Reading this suggests that artifacts are getting created more routinely than we might wish.
It happens with K1ii as well. I do not own other models.

05-03-2023, 01:00 PM - 1 Like   #5
Pentaxian
simon_199's Avatar

Join Date: Oct 2014
Posts: 613
I have not opened a ticket on the DxO support system. Maybe I could do it, and if a few more users would follow suit we could reach some "critical mass" that would make DxO more willing to look into the problem...

If anything could be done. Because we can't rule out the possibility that the lost detail is irrecoverable. I think I will never understand why Pentax went down this route of aggressive raw noise reduction. Nobody does that to this extent and there is absolutely no benefit. It should be a jpeg only setting.
05-03-2023, 01:39 PM   #6
Moderator
Loyal Site Supporter




Join Date: Feb 2015
Location: Central Florida
Photos: Gallery
Posts: 6,092
QuoteOriginally posted by simon_199 Quote

If anything could be done. Because we can't rule out the possibility that the lost detail is irrecoverable. I think I will never understand why Pentax went down this route of aggressive raw noise reduction. Nobody does that to this extent and there is absolutely no benefit. It should be a jpeg only setting.
Point one: EOS R3 uses what appears to be equally effective, or what you would call aggressive, noise reduction along the entire ISO range. Some photographers claim Canon's camera must logically be eating detail. I'd call it assumption.
https://www.dpreview.com/forums/post/65858159

Now is it possible detail might be lost in the process? Of course it's possible. Is it quantifiable? I've never seen it, and definitely not in any of my own images. Have Canon R3 users been able to prove detail is lost or even demonstrate a negative effect on its photos? I don't believe they have either. If it were so obviously true why has no one unequivocally shown it? Assumption that something must be happening is way short of proof.

Point 2: In my personal opinion DxO may have issues with out-of-the-ordinary images. They don't like monochrome images either whether from a Leica or my Pentax monos, and have no intention of changing that, They could. It does not surprise me that they would also have issues with a camera employing raw noise reduction.

Yet Topaz DeNoise handles K3III files fine, only rarely unexpected artifacts as far as I can see. I regularly use it. My On1 noise reduction, which gets used less often but still occasionally, has not given me any reason to distrust it. Not seeing any weird artifacts there either.

Is it possible that DxO is the one at fault? Of course it's possible. Neither Topaz nor On1 appears to have the same tendencies.

Last edited by gatorguy; 05-03-2023 at 07:51 PM.
05-03-2023, 04:46 PM - 5 Likes   #7
Site Supporter
Site Supporter
thix2112's Avatar

Join Date: Jan 2013
Location: North Alabama
Posts: 237
QuoteOriginally posted by SrMi Quote
The following are my observations.

AI-based noise reduction is a very powerful tool to eliminate or reduce noise.
Adobe has just launched its own AI Denoise, but unfortunately, it does not work with K-3 III DNGs (no NR applied).
DxO's PureRAW and PhotoLab 6 contain two relevant options. Unfortunately, the newer DeepPRIMEXD seems to generate artifacts, while DeepPRIME works well.

I hope this is helpful information.
I just sent a sample K3iii DNG to Adobe for review. I verified that Adobe Denoise AI did nothing to reduce noise in the K3iii DNG. They reached out to me to ask for a sample.

Tom

05-03-2023, 05:06 PM - 1 Like   #8
Moderator
Loyal Site Supporter




Join Date: Feb 2015
Location: Central Florida
Photos: Gallery
Posts: 6,092
QuoteOriginally posted by thix2112 Quote
I just sent a sample K3iii DNG to Adobe for review. I verified that Adobe Denoise AI did nothing to reduce noise in the K3iii DNG. They reached out to me to ask for a sample.

Tom
Excellent!
05-03-2023, 05:11 PM - 3 Likes   #9
Loyal Site Supporter
Loyal Site Supporter




Join Date: Jul 2021
Location: SF Bay Area
Posts: 45
Original Poster
QuoteOriginally posted by thix2112 Quote
I just sent a sample K3iii DNG to Adobe for review. I verified that Adobe Denoise AI did nothing to reduce noise in the K3iii DNG. They reached out to me to ask for a sample.

Tom
Great, thank you. I forgot to mention that I also filed a bug and sent a file earlier, but that was part of prerelease testing.
05-03-2023, 07:26 PM   #10
Loyal Site Supporter
Loyal Site Supporter
UncleVanya's Avatar

Join Date: Jul 2014
Photos: Gallery | Albums
Posts: 28,401
QuoteOriginally posted by gatorguy Quote
Point one: EOS R3 uses what appears to equally effective, or what you would call aggressive, noise reduction along the entire ISO range. Some photographers claim Canon's camera must logically be eating detail. I'd call it assumption.
Canon EOS R3 Baked in Raw Noise Reduction Revisited: Canon EOS R Talk Forum: Digital Photography Review
https://www.photonstophotos.net/Charts/PDR.htm#Canon%20EOS%20R3,Pentax%20K-1...ax%20K-3%20III

That's true indeed.
05-03-2023, 08:03 PM   #11
Moderator
Loyal Site Supporter




Join Date: Feb 2015
Location: Central Florida
Photos: Gallery
Posts: 6,092
Canon has been baking noise reduction in raw files at certain ISO"s for awhile, with the 1Dx III being the earliest I'm aware of back in 2016, about the same time Pentax began using it, and continuing with the mirrorless R5 and R6. The flagship R3 is the first from them with raw noise reduction baked in at all ISO's AFAIK.
05-03-2023, 08:23 PM - 1 Like   #12
Loyal Site Supporter
Loyal Site Supporter




Join Date: Jul 2021
Location: SF Bay Area
Posts: 45
Original Poster
QuoteOriginally posted by gatorguy Quote
Point 2: In my personal opinion DxO may have issues with out-of-the-ordinary images. They don't like monochrome images either whether from a Leica or my Pentax monos, and have no intention of changing that, They could. It does not surprise me that they would also have issues with a camera employing raw noise reduction.
The AI Denoising tools of DxO and Adobe are designed to work with Bayer or X-Trans patterns. The monochrome sensors do not have those patterns (no CFA) and therefore will likely never be supported.
05-03-2023, 08:27 PM   #13
Loyal Site Supporter
Loyal Site Supporter
UncleVanya's Avatar

Join Date: Jul 2014
Photos: Gallery | Albums
Posts: 28,401
QuoteOriginally posted by SrMi Quote
The AI Denoising tools of DxO and Adobe are designed to work with Bayer or X-Trans patterns. The monochrome sensors do not have those patterns (no CFA) and therefore will likely never be supported.
It does seem like they could if they wanted to - but the training might be a lot of effort for a small number of cameras.
05-03-2023, 08:28 PM   #14
Loyal Site Supporter
Loyal Site Supporter




Join Date: Jul 2021
Location: SF Bay Area
Posts: 45
Original Poster
The R3's NR is likely not as strong as Pentax's. In this graph, the data would not go below zero unless considerable NR is applied:

https://www.photonstophotos.net/Charts/RN_e.htm#Canon%20EOS%20R3_14,Pentax%20K-3%20III_14

However, as many have said, it is not an issue unless we notice problems in practice. On the other hand, I do not think there has been any serious investigation whether we lose anything because of NR.

---------- Post added 05-03-23 at 08:30 PM ----------

QuoteOriginally posted by UncleVanya Quote
It does seem like they could if they wanted to - but the training might be a lot of effort for a small number of cameras.
The AI denoising algorithms need un-demosaiced data.
05-03-2023, 08:49 PM   #15
Pentaxian
simon_199's Avatar

Join Date: Oct 2014
Posts: 613
Pentax is far more aggressive, at iso 400 or has a 1 stop advantage over a full frame sensor. Without any trickery an apsc sensor should be one stop behind (ballpark). So canon might apply some raw denoise, but Pentax is at a physics-defying level. K-3iii exceeds the performance of an ideal FF sensor according to that metric.

Software like Toapaz would be less sensitive of course because it is designed to work on finished RGB images of any sort, it is in a certain sense more generic. And more likely to yield bad results on any other camera as well, it is very image dependent. On the contrary if anybody has used DxO Deep Prime or XD he knows that on most cameras it is absolutely rock solid. You can set it at 20% (to be safe) at any iso, always on, without ever looking at it, and the likelihood of getting best-in-class results is extremely high. Unfortunately XD is broken for the K-3iii. But it works acceptably (at least improves on DP, and the slider does something) on other Pentax cameras like the KP, so it is definitely possible to fix it to an extent.


https://photonstophotos.net/Charts/PDR.htm#Canon%20EOS%205D%20Mark%20IV,Cano...ax%20K-3%20III

Last edited by simon_199; 05-03-2023 at 09:15 PM.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
adobe, aps-c, cameras, data, dslr, dxo, iii, iso, k-3, k-3 iii, k-3 mark 3, k-3iii, k1ii, lightroom, noise, nr, output, pentax, reduction, results, sensor, xd

Similar Threads
Thread Thread Starter Forum Replies Last Post
Topaz Labs JPEGtoRAW AI - essentially replaced by GigaPixel AI ?! BigMackCam Digital Processing, Software, and Printing 10 05-17-2022 01:34 AM
Camera Raw does not support K3 III PEF Files Fitz Pentax K-3 III 5 05-12-2021 12:04 PM
Applying smart effects to raw CarlJF Pentax Q 8 07-22-2015 04:40 PM
DPI of raw files vs jpg files MADDSNIPER Troubleshooting and Beginner Help 2 09-14-2013 11:37 AM
Can Nikon Ai and/or non Ai lenses be adapted to work on the K mount? Vantage-Point Canon, Nikon, Sony, and Other Camera Brands 3 07-28-2013 08:25 PM



All times are GMT -7. The time now is 10:33 AM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top