Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
07-23-2009, 01:31 AM   #1
Veteran Member




Join Date: Jul 2008
Location: Var, South of France
Photos: Albums
Posts: 1,071
How to use the K20's black pixels hidden in the DNGs...

Hello,

Well, for those wondering what is the difference between DNG and PEF, I've just found an interesting application to the extra info embedded in the DNG...

As you may be aware, DNGs generated by the K20 have the whole sensor in them, not just the exposed part... This includes the masked (not exposed) pixels used to evaluate black noise levels, located right at the edges of the sensor.

So, if you run RecoverEdges on them, you'll end up with black borders on the bottom and the right of the picture...

Now, you must also know about the ugly purple coloring that appears with high isos... On the k20, it is unevenly spread, and is more visible on the top and bottom of the pics (or left and right in a portrait orientation):



Now a real-world sample:

(just look at the hairs, left and right, and you'll see the purple noise creeping in...)

Now, by using these black pixels, maybe we could lessen this purple noise???
I gave it a try, and retrieved this black stripe, gaussian-blurred it (40 pixels settings), expanded it to the final image dimensions, and subtracted it from the picture above... Nothing more...



I find this waaaaaay better than the picture above...

Another try :

Original:


"Black noise" removed:


I'm pretty thrilled by the results! The only drawback of high iso on my K20 was these purple tops and bottoms (chroma noise is not really bothering me), and it seems there is a simple way to remove them!!!
Now, when can we expect a "black noise removal" tool in our RAW softwares???


Last edited by dlacouture; 07-23-2009 at 01:43 AM.
07-23-2009, 05:26 AM   #2
Veteran Member
krypticide's Avatar

Join Date: Dec 2007
Photos: Gallery
Posts: 1,079
Very interesting! I hate the purple noise too. Could you post a more detailed step-by-step of what you did?
07-23-2009, 05:49 AM   #3
Pentaxian




Join Date: Jun 2009
Location: GMT +10
Photos: Albums
Posts: 10,578
Good work. But ughh - that purple fringing at high ISO - what a drag. I daresay the K7 has the same issue.

That software looks useful. Downloading now...
07-23-2009, 07:24 AM   #4
Senior Member




Join Date: Aug 2007
Location: Perth, Australia
Posts: 268
I'm also interested to know how you select the black pixel for subtraction. Those results look really good.

07-23-2009, 07:29 AM   #5
Veteran Member




Join Date: Jun 2008
Location: Borås, Sweden
Photos: Gallery
Posts: 3,165
How is this any different from just blurring out some purple and subtracting that? Or just pulling the color purple back in post processing?
07-23-2009, 07:48 AM   #6
Veteran Member




Join Date: Jul 2008
Location: Var, South of France
Photos: Albums
Posts: 1,071
Original Poster
Okay, here is what I have done so far...
Please bear in mind that it's more a "proof of concept" than anything else... There is a huge place for improvement here... For instance, I'm not sure this will work with noise reduction applied by the RAW converter...

Vertical and Horizontal references below apply to a horizontal picture, and refer to the borders).

1 - on the K20 : shoot DNG!

2 - use RecoverEdges on a copy of your DNG (better keep the original DNG, as otherwise you'll have to do with the black pixels being visible, that means crops and so on...).

3 - Opening this DNG, crop the vertical black stripe only (the smallest dimension) and open it in your favorite pic editor.
Note :
- crop the same height than the picture, do not crop across the bottom black stripe.
- Do not include the white line, or the perfect-black stripe beyond!
- Do not use noise reduction!

4 - do a gaussian blur with a large enough setting as to remove any "blob" across the width (settings = around 40pixels in photoshop). This step could be improved as we only have to average along one dimension (width), but I could not find a ready-to-use tool...

5 - resize the resulting strip's width to your final photo size (should be 4687 pixels).

6 - open the original DNG (with or without NR, up to you), and add the processed image as a layer on top of your photo.

7 - set the mode to "Difference".

I'll try several things in the days to come, so maybe this will be updated soon!
07-23-2009, 07:51 AM   #7
Veteran Member




Join Date: Jul 2008
Location: Var, South of France
Photos: Albums
Posts: 1,071
Original Poster
Pingflood: the problem is that the purple noise addressed here is not uniformly spread across the frame, so you cannot just handle this through the WB tool... And reducing purple often yield strange results, especially with faces...
07-23-2009, 07:55 AM   #8
Veteran Member




Join Date: Jun 2008
Location: Borås, Sweden
Photos: Gallery
Posts: 3,165
OK, I see where you are going... does each shot have different amounts of purple or could you create one overlay image and some type of photoshop action to apply the change to future shots?

07-23-2009, 07:55 AM   #9
Veteran Member




Join Date: Jul 2008
Location: Var, South of France
Photos: Albums
Posts: 1,071
Original Poster
In case you wonder about the green tint in the processed pics above, by the way: I've made a custom WB on the purple-ridden pic, so when I subtracted the purple base noise, the whole pic took a slight green cast...

If you use predefined WB, the results would be much better, as you would not take this base purple noise into account when setting WB.
07-23-2009, 08:00 AM   #10
Veteran Member




Join Date: Jul 2008
Location: Var, South of France
Photos: Albums
Posts: 1,071
Original Poster
Well, it's really tied on how you processed your pic, regarding Exposure/Brightness/Contrast...

I would even say that it's linked to the physical properties of the sensor and surrounding electronics at the time the photo is made : temperature, time since switched on, etc...

So, no, a general template would not work IMO...

In fact it's really nearly what dark frame subtraction does... But in one shoot.
07-23-2009, 02:44 PM   #11
Pentaxian
Class A's Avatar

Join Date: Aug 2008
Location: Wellington, New Zealand
Posts: 8,934
QuoteOriginally posted by pingflood Quote
How is this any different from just blurring out some purple and subtracting that?
Just to clarify a bit more. The non-uniformity of the purple noise is mainly in the vertical direction.
The black stripe, which is normally not visible, captures this non-uniformity of the purple noise. Hence subtracting this pattern (widened to fit the whole image) should yield a better colour correction than subtracting the same amount of purple globally.

However, I have to say that the corrected versions do look a bit like globally corrected ones. Perhaps the effect would be clearer if one compared a globally corrected image to a "purple noise pattern"-corrected image?

QuoteOriginally posted by dlacouture Quote
In fact it's really nearly what dark frame subtraction does...
Yes, but using extrapolation of the noise pattern in one region to the whole image, rather than subtracting pixel for pixel.

Have you tried using the horizontal stripe as well? By creating the subtraction image by multiplying the vertical stripe with the horizontal one you may be able to better approach the real noise pattern. What I'm suggesting is like your widening of the vertical stripe, but instead of using the same weight everywhere across the frame, you could weight it with the value of the horizontal stripe for each horizontal position.
07-24-2009, 03:57 AM   #12
Veteran Member




Join Date: Jul 2008
Location: Var, South of France
Photos: Albums
Posts: 1,071
Original Poster
QuoteOriginally posted by Class A Quote
Have you tried using the horizontal stripe as well? By creating the subtraction image by multiplying the vertical stripe with the horizontal one you may be able to better approach the real noise pattern. What I'm suggesting is like your widening of the vertical stripe, but instead of using the same weight everywhere across the frame, you could weight it with the value of the horizontal stripe for each horizontal position.
I've thought about it, but as the purple noise seems quite one-dimensional, I think it would not really lead to better results... It must be tied with how the CMOS is handled...

I'll try to post pics with a global purple correction...
07-24-2009, 06:09 AM   #13
Veteran Member
WheresWaldo's Avatar

Join Date: Sep 2006
Location: North Carolina
Photos: Gallery
Posts: 498
This is extremely interesting dlacouture I am subscribing to this thread, otherwise I have nothing of value to add to the conversation.
07-25-2009, 10:58 AM   #14
Veteran Member
krypticide's Avatar

Join Date: Dec 2007
Photos: Gallery
Posts: 1,079
Okay, here's my try. These were shot at 6400, exposed to the right. The vertical stripes are masked parts of the image for comparison purposes.

15.0 pixel Gaussian:



40.0 pixel Gaussian:



You may have to click each image, right-click the image, and select "View Image" (Firefox, not sure what it is in Internet Explorer) in order to see it at 100% size.
08-30-2009, 12:10 PM   #15
Veteran Member




Join Date: Jul 2008
Location: Var, South of France
Photos: Albums
Posts: 1,071
Original Poster
Okay, a little update for those interested: this subject raised some serious interest on the "other" forum, and this resulted in a neat little application (still under development) capable of converting a straight-out-of-k20 DNG into a magenta-corrected version (you end up with a second DNG file).

Although still in a beta phase (doesn't handle compressed DNGs or PEF yet), it's really good and works quite marvels on high-iso pics...

Time to boost your auto-iso to 6400!

It's here:
K20 masked edges
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
bottom, camera, dslr, k20, noise, photography, picture, pixels, sensor, try
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
People hidden devisor Post Your Photos! 10 09-04-2010 12:49 AM
Is it me or ACR is really sluggish to precess K7's DNGs? alexeyga Digital Processing, Software, and Printing 2 05-28-2010 12:36 PM
K20 Hot Pixels, what to do? rdrum76 Pentax DSLR Discussion 12 01-06-2009 08:25 AM
Have to map stuck pixels on your new K20? Igilligan Pentax DSLR Discussion 13 12-23-2008 10:47 PM
Hidden d.bradley Post Your Photos! 6 06-29-2008 03:49 AM



All times are GMT -7. The time now is 01:14 AM. | See also: NikonForums.com, part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top