[ewelot]

Hi, everybody!

I just took my first images with a new K5IIs and I like it a lot. The main reason to buy this cam is to do some astrophotography for which its sensor is going to outperform my current Canon DSLRs.

The usual procedure to get nice deep-sky images is to stack many long exposures and process them using calibration images (mainly dark images and flat field exposures). So I started to get some dark images (lens cup on, camera in a dark room) using both manual exposure mode (up to 30 seconds) and bulb exposures of different exposure times. Then I had a look at the individual histograms of the grey raw data - without scaling and interpolation. To my surprise I get two different results:
- either a gaussian distribution around a mean value of about 512 ADU
- or half of a gaussian distribution, where the mean value has been shifted to 0 ADU, therefore cutting off the lower side of the gaussian.

Indeed, when looking at exif tags of the raw files there are different blackpoints used:
- all exposures in bulb mode (even at shortest exposure times) have blackpoint = 0
- exposures in manual mode with exposure time > 10 seconds have blackpoint = 0
- exposures in manual mode with exposure time <= 10 seconds have blackpoint near 512.
The first two cases will never allow to capture and subtract dark frames which is rather bad news for my goals.

I'm running firmware 1.0.2. All noise reduction has been disabled.

Is this intended behaviour (similar to Nikon's bad habit of pre-processing "raw" images)?
I really hope there is a workaround.
-
Thomas

[stevebrot]

Hi, everybody!

I just took my first images with a new K5IIs and I like it a lot. The main reason to buy this cam is to do some astrophotography for which its sensor is going to outperform my current Canon DSLRs.

The usual procedure to get nice deep-sky images is to stack many long exposures and process them using calibration images (mainly dark images and flat field exposures). So I started to get some dark images (lens cup on, camera in a dark room) using both manual exposure mode (up to 30 seconds) and bulb exposures of different exposure times. Then I had a look at the individual histograms of the grey raw data - without scaling and interpolation. To my surprise I get two different results:
- either a gaussian distribution around a mean value of about 512 ADU
- or half of a gaussian distribution, where the mean value has been shifted to 0 ADU, therefore cutting off the lower side of the gaussian.

Indeed, when looking at exif tags of the raw files there are different blackpoints used:
- all exposures in bulb mode (even at shortest exposure times) have blackpoint = 0
- exposures in manual mode with exposure time > 10 seconds have blackpoint = 0
- exposures in manual mode with exposure time <= 10 seconds have blackpoint near 512.
The first two cases will never allow to capture and subtract dark frames which is rather bad news for my goals.

I'm running firmware 1.0.2. All noise reduction has been disabled.

Is this intended behaviour (similar to Nikon's bad habit of pre-processing "raw" images)?
I really hope there is a workaround.
-
Thomas

Hi! Welcome to Pentax Forums!

How are you generating your histograms?

Do the blackpoint ADU values jive with output from ExifTool (or similar)?


Steve

(Not my normal area of expertise, but curious anyway...)

[ewelot]

Hi! Welcome to Pentax Forums!

How are you generating your histograms?

Do the blackpoint ADU values jive with output from ExifTool (or similar)?


Steve

(Not my normal area of expertise, but curious anyway...)

Thank you for your quick reply!

I'm using dcraw to convert from raw to linear 16bit tiff/ppm images in the so called "document mode" (-D switch). The histogram can then be viewed by any image processing software (e.g. I use ImageJ for this).

The blackpoint ADU values are embedded exif tags within PEF files. Should be visible by any tool which can display those tags. I use exiv2 for this purpose, but I run a Linux OS which is probably somewhat exotic amongst users of this forum. Anyway, no special tools. Should be possible to reproduce with standard tools on any system.
-
Thomas

[jbinpg]

I would be curious to see if your results stabilized if you had the viewfinder blackout clip on during the exposures. Theoretically, the mirror is up so there should be no light leakage. But theory and the real world are a long way apart at times.

Jack

[ewelot]

I would be curious to see if your results stabilized if you had the viewfinder blackout clip on during the exposures. Theoretically, the mirror is up so there should be no light leakage. But theory and the real world are a long way apart at times.

Jack

Yes, I forgot to mention that I had the viewfinder blackout clip on. Otherwise, you would indeed be fooled by scattered light entering the viewfinder.
-
Thomas

[jbinpg]

You don't by any chance have AWB on, do you? Much better to lock the WB on daylight for astro work.

Jack

[ewelot]

You don't by any chance have AWB on, do you? Much better to lock the WB on daylight for astro work.

Jack

Jack, that's another good point. But I already took it into account and locked the WB settings to daylight.
-
Thomas

[ChrisJ]

No way am I experienced in Astro Photography, but I wonder if using DNG rather than PEF makes a difference? DNG is uncompressed, all the files are the same size regardless of content, so it may be the Raw data without any interference from the camera.

When taking the dark frame don't you only need to record the noise in the sensor due to the heat generated when taking the stack of images? All I do is set the camera to Manual Mode, if it's not there already, place the lens cap on and alter the shutter speed to 1/60th of a second and take my dark frame. This will record a frame with the sensor noise in it which the software subtracts from the stack. Needless to say it has to be taken before the sensor cools down, usually within seconds of taking the stack of images.

Chris

[ewelot]

No way am I experienced in Astro Photography, but I wonder if using DNG rather than PEF makes a difference? DNG is uncompressed, all the files are the same size regardless of content, so it may be the Raw data without any interference from the camera.

When taking the dark frame don't you only need to record the noise in the sensor due to the heat generated when taking the stack of images? All I do is set the camera to Manual Mode, if it's not there already, place the lens cap on and alter the shutter speed to 1/60th of a second and take my dark frame. This will record a frame with the sensor noise in it which the software subtracts from the stack. Needless to say it has to be taken before the sensor cools down, usually within seconds of taking the stack of images.

Chris

I've got a few DNGs as well. But it makes no difference. Obviously, it is pre-processing before generating the raw file data.
By the way, DNG in Pentax K-5IIs is compressed. Adobes new DNG Standard now even allows DNG with lossy compression.

A few words about what I expect from dark frames and why I need the original, unaltered raw data for a proper subtraction.
If you are taking a bunch of dark images from CCD or CMOS sensors and look at an average master image you will most likely notice several distinct patterns:
- vertical stripes, due to the nature of the readout process
- dark current by thermically activated electrons proportional to exposure time and highly dependant on temperature, which might well be varying across the sensor (e.g. on Nikon D5200)
- cold, warm, hot pixels (anything in between)
- the usual statistical noise from dark current and readout

That picture is much different from a uniform dark level with just some noise added! There is a pattern which is equally present in both dark and light exposures - not in a single exposure but in the average of e.g. 100 shots. Therefore a simple subtraction of a constant dark level (blackpoint) is not appropriate.

I always found the dark patterns to appear fairly stable for a given camera at a fixed ISO, temperature and exposure time. Therefore I can capture dark images to produce a master dark before or after doing night sky exposures - even a few days later.

Would you like me to post sample master dark images (from another cam) to make the above more obvious?

I know that all this is not much of interest for common use of a DSLR. But I have the feeling that not keeping the original raw data as captured by the camera is something strange and in my opinion really unnecessary.

Thomas

[smigol]

Thomas,

I am extremely curious about your results. I use a K10D for astrophotos and ran across something similar when calibrating my images. The process I use to calibrate is via Maxim DL and I add a 300 ADU floor to the data. This ensures that the blackpoint is non-negative. FWIW, I do see individual values that are below 300 showing that even with calibration and adding the shoulder there is the possibility of dead pixels causing weirdness.

I assume that you are aware of Craig Stark's (of PHD) work on Canon's adjustment of the black point values in RAW. He did extensive testing of this behavior and published the results on his website. PDF Here

It seems that Pentax is doing something similar.

Stephen

[ewelot]

Thomas,

I am extremely curious about your results. I use a K10D for astrophotos and ran across something similar when calibrating my images. The process I use to calibrate is via Maxim DL and I add a 300 ADU floor to the data. This ensures that the blackpoint is non-negative. FWIW, I do see individual values that are below 300 showing that even with calibration and adding the shoulder there is the possibility of dead pixels causing weirdness.

I assume that you are aware of Craig Stark's (of PHD) work on Canon's adjustment of the black point values in RAW. He did extensive testing of this behavior and published the results on his website. PDF Here

It seems that Pentax is doing something similar.

Stephen

Stephen, thanks a lot for joining this thread! And congrats for your nice astro pics I've seen elsewhere in this forum.
I know about Canon's adjustment of the black point value from my own testing but I was not aware of Craigs article. Many thanks for pointing me to it! It fits extremly well to my findings. Obviously all camera manufacturers alter the original detected data counts. It is fine as long as the dark pattern and noise can be deduced. A simple adjustment of the average level to lower values is not a problem if information of most pixels is kept (this is the case in Canon's DSLRs).

But the sad point is: if you shift the black point level to zero then 50% of all sensor pixels in a dark exposure have NO information at all.
How is your K10D doing in this respect?

I'm preparing a few graphs from darks captured using my K5IIs for illustration purposes right now. Will be posted soon.

Thomas

[russell2pi]

I might be way off target here, but could this be due to in-camera High ISO Noise Reduction? If you have it on the auto setting it seems to kick in at ISO 1600 and above. It takes a dark frame after every exposure, even in RAW mode, so I've been assuming it actually subtracts the dark frame before writing the RAW file.

[jbinpg]

That indeed would be true if High ISO noise reduction were on. But the whole idea behind stacking is to reduce noise so every astro photographer I know turns off any in-camera noise reduction precessing. I would assume the OP has no noise reduction turned on in his camera.

Jack

[smigol]

Thomas,

Regarding the K10D, I've decided to use ISO 100 with 1200 sec subs as my default setting. When I used the ISO setting before I started calibrating with Maxim, I was disappointed in how much of the dark detail was missing in my shots. For a year, I used ISO 400 with 600 second durations to get the data out of the lowest ranges. After re-running the numbers on the camera, I decided to go back to ISO 100 and use much longer subs. This has given the effect of getting the signal out of the lowest areas while leaving a maximum amount of headroom *plus* I'm adding a shoulder of 300 ADU in Maxim.

This combination has finally allowed me to get the amp glow in the K10D under control. No other calibration routine seemed to properly handle the bad areas.

At this point, if Pentax does do funny stuff with the data before it's written to the RAW file, at least I have only one variable to manage: temperature. Everything else has been held constant. Also, the K10D is a CCD based camera, so some of the noise reduction requirements are not required like with CMOS devices.

Someday I'll have to see if I can reconcile my camera's numbers with Craig's method.

Everytime I think if you can trick the K5IIs into not clipping the blacks by using a hot environment it becomes moot because you would not have the same environment in the field. Thus, the camera would clip.

What do your bias files look like? Are they clipped?

[ewelot]

@russell2pi, jbinpg

Yes, I turned off any high iso noise reduction and long exposure noise reduction features. Maybe it was not stated clearly in my earlier posts.

Thomas,

Regarding the K10D, I've decided to use ISO 100 with 1200 sec subs as my default setting. When I used the ISO setting before I started calibrating with Maxim, I was disappointed in how much of the dark detail was missing in my shots. For a year, I used ISO 400 with 600 second durations to get the data out of the lowest ranges. After re-running the numbers on the camera, I decided to go back to ISO 100 and use much longer subs. This has given the effect of getting the signal out of the lowest areas while leaving a maximum amount of headroom *plus* I'm adding a shoulder of 300 ADU in Maxim.

This combination has finally allowed me to get the amp glow in the K10D under control. No other calibration routine seemed to properly handle the bad areas.

At this point, if Pentax does do funny stuff with the data before it's written to the RAW file, at least I have only one variable to manage: temperature. Everything else has been held constant. Also, the K10D is a CCD based camera, so some of the noise reduction requirements are not required like with CMOS devices.

Someday I'll have to see if I can reconcile my camera's numbers with Craig's method.

Everytime I think if you can trick the K5IIs into not clipping the blacks by using a hot environment it becomes moot because you would not have the same environment in the field. Thus, the camera would clip.

What do your bias files look like? Are they clipped?

I don't think you could trick the camera by using a hot environment. I guess the in-camera pre-processing routines analyze the data from unexposed sensor area (overscan area, which is on the left side in the K5IIs) to determine some kind of average count. This value - which depends on temperature - is then subtracted from the whole image before writing the raw data to file resulting in approx. 50% clipped pixels in any dark exposure.



For a better understanding of the black point issue I'll now present a few results obtained with my K5IIs at ISO 800, 10 seconds exposure time, 0°C air temperature under complete darkness.

1. Master dark (average of 30 dark exposures) in manual mode.



The image above shows the whole sensor area. You can clearly see vertical stripes and some amp glow or heat source at top and right.
This is a very typical and perfectly usable dark image for a low dark signal environment (cold, short exposure in terms of astrophotography).


2. Histogram plots (from a 4000 pixel area) of a single exposure.

From left to right:
- exposure in manual mode
- exposure in bulb mode
- exposure in bulb mode with the bin at 0 ADU omitted (for better comparison to the first one).


The effect of clipping the left side of the gaussian noise (about 50% of all pixels) in the bulb mode is shown. The original black point near 512 ADU is preserved only by exposures in manual mode and at 10 seconds and below.


3. Average Row (from master dark images comprising of 30 individual exposures)

The vertical dark image pattern as seen in 1. is measured by averaging many rows. The left figure shows ADU vs. column number for the master dark obtained in manual mode, the right figure shows similar measurements (same sensor area) using images in bulb mode.


It is very obvious that the pattern visible in unbiased dark images (mainly columns with lower ADU, visible in left figure) has vanished in the noise-clipped bulb mode images to a large degree! But it will appear in any light exposure because counts are much higher then and no clipping happens. The goal of removing the underlying dark pattern by subtractiing a master dark image can not be achieved anymore for bulb exposures.

Another interesting effect is that by averaging noise-clipped images we get a mean value above zero! This will definitely affect image reduction when it comes to vignetting correction (known as flat field reduction). When you model this effect then it appears that darks with higher intrinsic noise will result in a higher "false mean". This might well account for Stephen's observation that shooting at higher ISO will make dark frames even more inappropriate. There is both missing information and statistics is fooling you too.


What makes me burning my head is: Why does Pentax apply this shift of the black level to zero? If it were to ease the average user in processing their raw images then they would have implemented this shift for all exposure settings or at least for the most common setups which are with short exposure times. But Pentax engineers have decided to do the reverse ...

Any thoughts?

Thomas