Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
03-30-2009, 02:21 PM   #16
Veteran Member
Ben_Edict's Avatar

Join Date: Jul 2007
Location: SouthWest "Regio"
Photos: Albums
Posts: 3,303
QuoteOriginally posted by Confused Quote
Yet LeoTaylor writes:

What I fail to comprehend fully in this instance is how on earth the dark frame can be subtracted later, having taken the initial photo with a K10D ? Surely the dark frame is an integral part of selecting the Noise Reduction feature ? Sorry if I am missing the blindingly obvious here, but would anyone care to clarify this matter for me ?

Best regards
Richard
Richard, darkframe subtraction is a technique that has been around since the advent of digital in astronomy about 20 years ago. You simply make a second image with the same settings as the initial exposure of your subject, but during the darkframe exposure you keep the cap on the lens (aka telescope). With the help of computer software you can later subtract this darkfram from the image to reduce sensor-inherent noise.

To simplify life, astronomers tend to collect dark frames, made with varying lengthes and under different ambient temperatures. You can either use these for any matching exposure. Or you can be much more elaborate and combine different dark frames (averaging) to generate a master darkframe, which is not only better, but more universal too.

The in-camera darkframe subtraction is a late invention and builts upon the experience we have in astronomy for years.

Ben

03-30-2009, 05:14 PM   #17
Site Supporter
smigol's Avatar

Join Date: Sep 2008
Location: Menlo Park, CA
Photos: Albums
Posts: 391
So a master dark frame is something that is independent of temperature and exposure duration? If so, then I can be building a library of dark frames right now!

Any idea of what a good sample size should make a proper master dark frame?
03-30-2009, 07:30 PM   #18
Inactive Account




Join Date: Jul 2007
Location: London
Posts: 1,067
Hi Ben

Kindly forgive any unintentional ambiguity in the way that I expressed my original query…..! I believe I already comprehend the underlying principle of dark-frame subtraction, so I'll take a fresh stab at the matter and word things slightly differently.
Let's say for example that I intend taking a 30-second night exposure using my K10D and choose to employ it's Noise Reduction feature. In this instance the minimum time that would pass before I could attempt to take a further shot would be 30 seconds exposure + 30 seconds (dark-frame subtraction process) making a grand total of at least 60-seconds, during which the majority of thermal noise has hopefully been subtracted.
My point is that although initially separate, these two stages (exposure + dark-frame subtraction process) form a sequential chain of events, producing a merged image. To the best of my knowledge, there is no way in which "the dark-frame subtraction can be handled later" because it's already taken place ! Or am I still missing something ?

Best regards
Richard
03-30-2009, 07:43 PM   #19
Veteran Member
Robert S Donovan's Avatar

Join Date: Mar 2009
Location: Anderson, SC
Photos: Gallery
Posts: 361
Original Poster
Richard,
I'll let Ben chime in on this when he is able since he knows a lot more about it than I do. The misunderstanding I am noticing in your last post is that the post process dark frame subtraction that Ben speaks of is replaces the in-camera dark frame subtraction. In other words, you turn off the long exposure Noise Reduction in your K10D and then, using the dark frames you took at another time, remove the noise later in post processing with your computer.

IIRC, the duration, ambient temperature and ISO of your dark frames should all match the long exposures that you wish to apply them to as closely as possible. Theoretically, done correctly, this post process method should be more accurate than the in-camera method since the dark frames taken by the camera are subject to greater heat build-up since they immediately follow the long exposure.

Still seems like a lot of work to me...

Cheers!

03-30-2009, 07:51 PM   #20
Inactive Account




Join Date: Jul 2007
Location: London
Posts: 1,067
Hi Robert

QuoteQuote:
the post process dark frame subtraction that Ben speaks of replaces the in-camera dark frame subtraction
Ah-ha, at long last the mist begins to clear a little.......

Best regards
Richard
03-31-2009, 01:30 AM   #21
Veteran Member
Ben_Edict's Avatar

Join Date: Jul 2007
Location: SouthWest "Regio"
Photos: Albums
Posts: 3,303
QuoteOriginally posted by smigol Quote
So a master dark frame is something that is independent of temperature and exposure duration? If so, then I can be building a library of dark frames right now!

Any idea of what a good sample size should make a proper master dark frame?
Master darkframes are party dark art and partly applied physics... I just read a very good article in a German astronomy magazine, which is probably not of much help for you. But here are some useful links for you:
Image Data Calibration
K3CCDTools deep sky processing tutorial (software specific, but useful)
LRGB CCD Image processing with Iris (software specific, but useful)
Telescope Reviews: Creating A Master Dark Frame Library (an informed discussion)

Unfortunately a master darkframe should also be used with images taken at the same or a similar temperature as the master darkframe itself.

The sense of the master darkframe is, to average several darkframes to reduce the effect of completely random noise, when you combine the darkframe with the raw-image. It could be, that a single darkframe contains excessive noise or random cosmics or whatever and if you use this darkframe you would not only reduce noise, but introduce artefacts into your image. Using an averaged master darkframe reduces this risk. This is also the reason, why you can use master darkframes over a somewhat extended period of time, as single variations are smoothed out.

Ben
03-31-2009, 01:32 AM   #22
Veteran Member
Ben_Edict's Avatar

Join Date: Jul 2007
Location: SouthWest "Regio"
Photos: Albums
Posts: 3,303
QuoteOriginally posted by Robert S Donovan Quote
Richard,
I'll let Ben chime in on this when he is able since he knows a lot more about it than I do. The misunderstanding I am noticing in your last post is that the post process dark frame subtraction that Ben speaks of is replaces the in-camera dark frame subtraction. In other words, you turn off the long exposure Noise Reduction in your K10D and then, using the dark frames you took at another time, remove the noise later in post processing with your computer.
Robert, your completely right. I simply did not realize, that Richard thought darkframe subtraction to be an additional stept after in-camera subtraction, which ofcourse is a valid assumption. Sorry, for that misunderstanding.

Ben
03-31-2009, 03:11 AM   #23
Senior Member




Join Date: Feb 2009
Location: west coast USA
Posts: 206
QuoteOriginally posted by StevenVH Quote
I've been wondering about both NR settings in general with my K200D. [...] I'd love to hear something definitive about the subject in general, especially if it applies to the K200D as well
Generally speaking, there are two types of noise in digital camera images. One is thermal noise from the sensor itself, which (aside from true "hot pixels") shows up only in longer exposures. With this type, a particular photosite (which in Bayer CFAs translates to a pixel of a single color) is artificially marked by heat rather than by actual light hitting it. As mentioned above, dark frame subtraction is used to deal with this noise; however, this technique works only because the thermal noise comes from the manufacturing characteristics of the sensor itself, and is therefore consistent, not random.

Since it is consistent, dark frame subtraction is relatively precise and doesn't remove data from the actual image. That means it can safely be applied to the RAW output of a camera without degrading image quality. Also, since a noise sample needs to be generated from the sensor itself, this kind of noise reduction can't accurately be done by a pure software algorithm; to do as good a job later, software needs a dark frame. That leaves choices of generating your own dark frames to subtract later, or simply letting the camera do it before you get the RAW output. Either way, this process generally improves image quality rather than degrading it, and normally is applied to the RAW file from the camera.

The other type is signal noise, which is inherent in the technology used in the entire sensing pipeline. A signal amplifier (used for ISO selection) both amplifies any noise inherent in the data from the sensor, and adds noise of its own, which is why higher ISOs are generally noisier. Decreasing this type of noise is of course something the hardware manufacturers battle with every year.

Signal noise is essentially random, so a surgical technique like dark frame subtraction doesn't work on it. Instead, software noise reduction algorithms attempt to figure out which part of the image data is actually random noise, and remove it. Because Bayer CFA type sensors only capture one color at each photosite, interpolation algorithms (used on the way to a full-resolution human-viewable image) are concerned with reconstructing detail in the scene that the sensor was unable to capture. Similarly, noise reduction tries to avoid removing detail along with the noise. Both of these processes use algorithms that need to find detail in the image, so generally noise reduction is applied during or directly after interpolation, in order to get the most clean yet detailed result possible.

This type of noise reduction is generally harmful to the image data, so ideally you want the raw data from the camera before it is performed. This allows more involved software with newer algorithms to work on the image later, and get a better result. Also, since it usually occurs after interpolation, the data noise reduction operates on is already "cooked" in some sense and it's not practical for the camera to turn around and save it as "raw" again. For these reasons, signal noise reduction is normally not applied to RAW output files.

(There are exceptions: I believe some of Sony's DSLRs apply signal noise reduction to RAW files under certain conditions, and some algorithms work better before interpolation is done.)


As for the K200D specifically, I did some tests under low light conditions, taking several images of this scene:



The marked areas are used in 100% crop comparisons below. All images were taken in RAW+JPEG mode. The raw images were processed with RAWHide, using ECW interpolation and the default 1.5 sharpening strength, mostly because I like how it renders the noise. (Note that RAWHide renders slightly blue compared to both the camera and PPL; I'm not sure why.) Final cropping and assembly was done with IrfanView.


ISO 1600, 1/6", f/4.5, changing only the High ISO Noise Reduction setting:



Two things are notable about the results: this noise reduction is not applied to the RAW output, and even with NR off Pentax is applying chroma noise reduction to the JPEGs. The NR choice appears to be about luminance, which is much more destructive to image detail. Always doing chroma NR seems warranted to me, given the blue static present without it.


For the second comparison, the reference image is ISO 100, 30", f/16. The two test images are ISO 400, 30", f/29. (I goofed on the exposure, so these two are +1/3 EV compared to the others.) High ISO NR is off, only Slow Shutter NR is changed:



This type of NR is applied to both the JPEG and RAW files. You can easily see the kind of noise this setting is meant to reduce, and it does quite well at it, although it's not perfect. Note again that something is reducing chroma noise in the JPEG results.

Compared with the reference image, it should also be apparent that dark frame subtraction can't help with the random signal noise at higher ISOs. It does change the grain slightly, but that's about it.

Hope this helps.


Last edited by Quension; 06-22-2012 at 06:56 PM. Reason: change in image hosting
03-31-2009, 05:07 AM   #24
Veteran Member
Robert S Donovan's Avatar

Join Date: Mar 2009
Location: Anderson, SC
Photos: Gallery
Posts: 361
Original Poster
@Quension: Nicely done!
03-31-2009, 11:46 PM   #25
Site Supporter
smigol's Avatar

Join Date: Sep 2008
Location: Menlo Park, CA
Photos: Albums
Posts: 391
QuoteOriginally posted by Ben_Edict Quote
Master darkframes are party dark art and partly applied physics... [snip]
Ben

Ben, Thanks for the links! I've read through them and I have a better idea of what I can do with my next imaging session. I'm getting closer to where I want to go with my K10D. It seems that I'll be planning to get a collection of 5 or so dark frames each time I go out for reference purposes.

Any ideas on a technique to create a flat field frame? I'm guessing that I'd need to find a way to evenly illuminate the front of the lens -- maybe a piece of cloth across the lens front and then placing it in the sunlight would do the trick?
04-01-2009, 01:19 AM   #26
Veteran Member
Ben_Edict's Avatar

Join Date: Jul 2007
Location: SouthWest "Regio"
Photos: Albums
Posts: 3,303
QuoteOriginally posted by smigol Quote
Ben, Thanks for the links! I've read through them and I have a better idea of what I can do with my next imaging session. I'm getting closer to where I want to go with my K10D. It seems that I'll be planning to get a collection of 5 or so dark frames each time I go out for reference purposes.

Any ideas on a technique to create a flat field frame? I'm guessing that I'd need to find a way to evenly illuminate the front of the lens -- maybe a piece of cloth across the lens front and then placing it in the sunlight would do the trick?
The flat field needs to be taken under similar conditions as the darkframe. So making it in bright sunlight won't be a good idea, as the camera gets too hot. Usually people build something like a light box (for sorting through slides or negatives), which they place in front of the lens. What should also work is something like a white balance front cap for the lens. The trick is, to have the illumination as even as possible. In out observatory we have a large light box, at which we aim the scopes to get the flat fields.

Ben
08-16-2013, 09:48 AM   #27
New Member




Join Date: Sep 2011
Location: Preston, Nr. Canterbury, Kent, England
Posts: 5
Having read this thread and only just started 2 minute exposures of some constellations, I have found that on my K10 the NR does indeed reduce speckling of the image at 100ISO where 2 minutes exposure occurs, but only to the benefit of the JPEG's it seems. Having now turned NR off and shooting RAW (plus a JPEG for reference) all the time, the RAW files don't seem to suffer for speckling as do the JPEG's tagged with them. I like RAW as I'm only taking pictures I want to use, develop, RAW allows that post cold hands session to tweak the best out the images taken, but the exposure at source is always going to be critical to getting the most from the RAW image too, so RAW's aren't the 'lets fiddle later' option, better you spend time getting things as close to correct exposure as possible at the start - and I'm still learning...given some of the blurry, speckled images I still keep creating now and then! Familiarity with controls is essentially for astro' pics, or else you won't take time to make those fine adjusts I believe, which is what the DSLR allows us.
08-16-2013, 10:55 PM   #28
Site Supporter
smigol's Avatar

Join Date: Sep 2008
Location: Menlo Park, CA
Photos: Albums
Posts: 391
Antzpad, the K10D has nice noise profiles, once you understand how they work. The amp glow is pretty fierce and hard to remove. Pentax did some trickery with data manipulation before RAW (so does Canon and Nikon) to get the most out of the dark frame subtraction process. It's this step that seems to give the in camera NR the better way to control the glow.

Critical bits I've learned over the years:
Use the lowest ISO setting you can get away with under the circumstances. At 12 bits, there's not a lot of bit depth to be wasteful. It also keeps the amp glow from saturating too much of the signal.
Try to keep the camera as cool as possible.
Gather a library of dark frames and sort by EXIF temperature for your own dark frame matching.
Your RAW converter may have the ability to map hot pixels. This is worth doing as removing these outliers will give better noise modeling.

The amp glow areas really show up when taking exposures longer than 5 minutes.

Good luck!
08-18-2013, 04:14 AM   #29
New Member




Join Date: Sep 2011
Location: Preston, Nr. Canterbury, Kent, England
Posts: 5
Thanks for the comprehensive response, it is appreciated. I have to say I don't completely understand the technical aspects of amp glow currently, but as my experience grows I hope to have a better grasp of how to achieve more consistent results. I am copying the reply for future reference. Thank you - Anthony.
08-19-2013, 07:17 AM   #30
Senior Member
insulinguy's Avatar

Join Date: Aug 2009
Location: Fort Worth, TX
Photos: Gallery | Albums
Posts: 134
i am a devoted K20 user but the DFS was killing me so i bought a K10 body just for night shooting. since you can turn DFS off i dont understand why the OP was complaining as the ability to turn it off is right there in the menu. clean up the noise in post and save yourself the beating of sitting there for 30 minutes waiting to take another shot. this is especially important when shooting spinning wool as you need about a 5 second exposure and the wool burns out in about 10-12 seconds so waiting around to shoot another shot means you just missed the other shot!
http://www.flickr.com/photos/polyurethanewheels/9456975778/
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
camera, dslr, exposure, exposures, k10d, noise, pentax k10d, photography, reduction
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
long exposure noise? slow2focus Troubleshooting and Beginner Help 9 12-16-2009 09:24 PM
K-x long exposure noise? er1kksen Pentax DSLR Discussion 4 11-06-2009 01:17 PM
Long exposures and in-camera noise reduction ChipB Pentax DSLR Discussion 3 04-12-2008 04:28 AM
K10D long exposure noise issue Smoothster Pentax DSLR Discussion 27 03-03-2008 04:01 AM
Noise Reduction and Raw Processing Kemal Pentax DSLR Discussion 10 09-06-2007 08:10 AM



All times are GMT -7. The time now is 08:46 AM. | See also: NikonForums.com, part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top