[dlacouture]

Hello,

Well, for those wondering what is the difference between DNG and PEF, I've just found an interesting application to the extra info embedded in the DNG...

As you may be aware, DNGs generated by the K20 have the whole sensor in them, not just the exposed part... This includes the masked (not exposed) pixels used to evaluate black noise levels, located right at the edges of the sensor.

So, if you run RecoverEdges on them, you'll end up with black borders on the bottom and the right of the picture...

Now, you must also know about the ugly purple coloring that appears with high isos... On the k20, it is unevenly spread, and is more visible on the top and bottom of the pics (or left and right in a portrait orientation):



Now a real-world sample:

(just look at the hairs, left and right, and you'll see the purple noise creeping in...)

Now, by using these black pixels, maybe we could lessen this purple noise???
I gave it a try, and retrieved this black stripe, gaussian-blurred it (40 pixels settings), expanded it to the final image dimensions, and subtracted it from the picture above... Nothing more...



I find this waaaaaay better than the picture above...

Another try :

Original:


"Black noise" removed:


I'm pretty thrilled by the results! The only drawback of high iso on my K20 was these purple tops and bottoms (chroma noise is not really bothering me), and it seems there is a simple way to remove them!!!
Now, when can we expect a "black noise removal" tool in our RAW softwares???

---

Last edited by dlacouture; 07-23-2009 at 01:43 AM.

[krypticide]

Very interesting! I hate the purple noise too. Could you post a more detailed step-by-step of what you did?

[rawr]

Good work. But ughh - that purple fringing at high ISO - what a drag. I daresay the K7 has the same issue.

That software looks useful. Downloading now...

[mcjm78]

I'm also interested to know how you select the black pixel for subtraction. Those results look really good.

[pingflood]

How is this any different from just blurring out some purple and subtracting that? Or just pulling the color purple back in post processing?

[dlacouture]

Okay, here is what I have done so far...
Please bear in mind that it's more a "proof of concept" than anything else... There is a huge place for improvement here... For instance, I'm not sure this will work with noise reduction applied by the RAW converter...

Vertical and Horizontal references below apply to a horizontal picture, and refer to the borders).

1 - on the K20 : shoot DNG!

2 - use RecoverEdges on a copy of your DNG (better keep the original DNG, as otherwise you'll have to do with the black pixels being visible, that means crops and so on...).

3 - Opening this DNG, crop the vertical black stripe only (the smallest dimension) and open it in your favorite pic editor.
Note :
- crop the same height than the picture, do not crop across the bottom black stripe.
- Do not include the white line, or the perfect-black stripe beyond!
- Do not use noise reduction!

4 - do a gaussian blur with a large enough setting as to remove any "blob" across the width (settings = around 40pixels in photoshop). This step could be improved as we only have to average along one dimension (width), but I could not find a ready-to-use tool...

5 - resize the resulting strip's width to your final photo size (should be 4687 pixels).

6 - open the original DNG (with or without NR, up to you), and add the processed image as a layer on top of your photo.

7 - set the mode to "Difference".

I'll try several things in the days to come, so maybe this will be updated soon!

[dlacouture]

Pingflood: the problem is that the purple noise addressed here is not uniformly spread across the frame, so you cannot just handle this through the WB tool... And reducing purple often yield strange results, especially with faces...

[pingflood]

OK, I see where you are going... does each shot have different amounts of purple or could you create one overlay image and some type of photoshop action to apply the change to future shots?

[dlacouture]

In case you wonder about the green tint in the processed pics above, by the way: I've made a custom WB on the purple-ridden pic, so when I subtracted the purple base noise, the whole pic took a slight green cast...

If you use predefined WB, the results would be much better, as you would not take this base purple noise into account when setting WB.

[dlacouture]

Well, it's really tied on how you processed your pic, regarding Exposure/Brightness/Contrast...

I would even say that it's linked to the physical properties of the sensor and surrounding electronics at the time the photo is made : temperature, time since switched on, etc...

So, no, a general template would not work IMO...

In fact it's really nearly what dark frame subtraction does... But in one shoot.

[Class A]

How is this any different from just blurring out some purple and subtracting that?

Just to clarify a bit more. The non-uniformity of the purple noise is mainly in the vertical direction.
The black stripe, which is normally not visible, captures this non-uniformity of the purple noise. Hence subtracting this pattern (widened to fit the whole image) should yield a better colour correction than subtracting the same amount of purple globally.

However, I have to say that the corrected versions do look a bit like globally corrected ones. Perhaps the effect would be clearer if one compared a globally corrected image to a "purple noise pattern"-corrected image?

In fact it's really nearly what dark frame subtraction does...

Yes, but using extrapolation of the noise pattern in one region to the whole image, rather than subtracting pixel for pixel.

Have you tried using the horizontal stripe as well? By creating the subtraction image by multiplying the vertical stripe with the horizontal one you may be able to better approach the real noise pattern. What I'm suggesting is like your widening of the vertical stripe, but instead of using the same weight everywhere across the frame, you could weight it with the value of the horizontal stripe for each horizontal position.

[dlacouture]

Have you tried using the horizontal stripe as well? By creating the subtraction image by multiplying the vertical stripe with the horizontal one you may be able to better approach the real noise pattern. What I'm suggesting is like your widening of the vertical stripe, but instead of using the same weight everywhere across the frame, you could weight it with the value of the horizontal stripe for each horizontal position.

I've thought about it, but as the purple noise seems quite one-dimensional, I think it would not really lead to better results... It must be tied with how the CMOS is handled...

I'll try to post pics with a global purple correction...

[WheresWaldo]

This is extremely interesting dlacouture I am subscribing to this thread, otherwise I have nothing of value to add to the conversation.

[krypticide]

Okay, here's my try. These were shot at 6400, exposed to the right. The vertical stripes are masked parts of the image for comparison purposes.

15.0 pixel Gaussian:



40.0 pixel Gaussian:



You may have to click each image, right-click the image, and select "View Image" (Firefox, not sure what it is in Internet Explorer) in order to see it at 100% size.

[dlacouture]

Okay, a little update for those interested: this subject raised some serious interest on the "other" forum, and this resulted in a neat little application (still under development) capable of converting a straight-out-of-k20 DNG into a magenta-corrected version (you end up with a second DNG file).

Although still in a beta phase (doesn't handle compressed DNGs or PEF yet), it's really good and works quite marvels on high-iso pics...

Time to boost your auto-iso to 6400!

It's here:
K20 masked edges