[smigol]

Thomas,

I did some testing of a K10D yesterday to see what happens with the black point and histogram at the 1, 5, 10, 15, and 30 second durations with manual and bulb exposures. No strict temperature controls applied at the time, it was just at ambient and didn't make the camera reach thermal equilibrium. Still, the temperatures are mostly stable because the durations were short. At 100 ISO, the blackpoints are all 0. The mean is uniformly .7 to .8 ADU. Max was due to hot pixels and min was due to dead pixels.

When stretching the histogram to show the full range, it looks more like your second example - a truncated curve with a tail leading off to the left with a spike.


The background bias noise in the above example has been stretched to show the histogram. The histogram is generated in the above view by averaging the 3x3 area around the mouse pointer which is placed at the center of the image.



This view above is the same sub with a 300% view. This shows the density of the bias noise. Note the location of the hot pixels as large blocks of color. Also note the lack of black pixels - maybe this is a view of properly non-truncated data?

I do see bias in the darks Thomas has posted (and the ones I generated of short duration), so it's the prominent source of noise as the vertical pattern suggests. The ring of glow around the edges in the K5IIs example looks like classic amp glow and matches what I saw in a dark that I've seen on a K5.

To answer the question of why did the engineers do this, I'd point to what Craig Stark says in his conclusion in the Canon study. It's to optimize the performance of the camera for the common daylight use. There is some tradeoff of dark signal being lost, maybe it's not a full stop like Stark found, but it is a sacrifice. The question is still raised, why is it not scaled under common exposures (less than 10 seconds) and then turned on by default at anything above? Thomas is right, this seems to be a reverse of what would be desired. Again, my guess is that the engineers may be thinking less about saving the dim data and more about saving headroom for highlights.

What's the remedy? A firmware update would be nice that gives a toggle to reset black point - that would be ideal. Maybe Pentax will give us this control if they did, bravo, if not, we may have to use workarounds of tricking the camera and standardizing on this behavior.

In practice, it seems that if the camera's performance is profiled at a particular exposure duration and ISO setting, a map of noise across a range of temperatures would generate a usable reduction master. As these cameras are not temperature regulated, we have to depend on the EXIF temp reading and matching darks to lights to develop a profile. In my experience, this linking of temperatures needs to be within 2 C to be usable. Anything more does not scale properly.

Once we decide on the setting that works for the conditions used (for me, I've said it's 1200 second subs at 100 ISO) then all the rest of the efforts revolve around that point. Mount tuning, alignment, library management, and power supply are all part of the effort to support this choice. Maybe these long durations are enough to get the heat signal out of the murk. Maybe not as even a 1200 second 29C sub I referenced had a black point of 0 and mean of 3 (median of 0).

[jbinpg]

As an aside to one of your points, I have found that the time and temperature of a long exposure as written to the EXIF is that of the start of the exposure not the end. Have you also found that?

Jack

[smigol]

I see that, too. The EXIF date taken value is the start and the file system "date" value is the time it was written. Easy to see with long exposures.
The temp data seems to be the same as well. Easy to test, too. Refrigerate the camera, start a long exposure, put it in a warm environment so that during the exposure it heats up. The EXIF temp will be the cold value. Also, it's well known that the EXIF temp is from a sensor that is not close to the image sensor. It's someplace else and is only good for general use.

[ewelot]

Stephen,

just a small note about your measurements. From the images you posted I assume you measured the dark in RGB images. I'd suggest you do this in the gray data before they are de-bayered. Otherwise - regardless of the algorithm used for de-bayering - data are interpolated which heavily affects the histogram (e.g. the number of pixels found at ADU=0 decreases a lot due to de-bayer algorithm).
The same applies for dark image reduction: you should always subtract dark from light exposures before de-bayering. But maybe you did it already this way.

[tuco]

...
but I run a Linux OS which is probably somewhat exotic amongst users of this forum.

There are more using it than you might think.

My first installation of Linux was around 1994. I had it for a main desktop at one point when it got better but eventually the I had to get a different UNIX system that supported secret, proprietary file formats for video editing and could run more popular image editors that supported more RAW files and could handle 48 bit color. People using Linux today have it way-easy compared to back then.

[smigol]

Yes, I do all calibration before debayering. The screengrabs were from fastpictureviewer as it was quick and easy. I did my testing yesterday with dcraw generated tiffs with -D -r 1 1 1 1 -4 -T. Then I fed these into PixInsight to get the mean and median values.

K values ranged from .7 to .9 ADU for durations from 1/33 to 30 seconds. The 1200 second dark had a mean K value of 3 ADU.

I believe that the final answer is "Pentax is messing with the data before RAW" which isn't surprising.

[ewelot]

There are more using it than you might think.

My first installation of Linux was around 1994. I had it for a main desktop at one point when it got better but eventually the I had to get a different UNIX system that supported secret, proprietary file formats for video editing and could run more popular image editors that supported more RAW files and could handle 48 bit color. People using Linux today have it way-easy compared to back then.

I'm not alone anymore ...! (I started using Linux in 1992 - just for the courious.)

Yes, I do all calibration before debayering. The screengrabs were from fastpictureviewer as it was quick and easy. I did my testing yesterday with dcraw generated tiffs with -D -r 1 1 1 1 -4 -T. Then I fed these into PixInsight to get the mean and median values.

K values ranged from .7 to .9 ADU for durations from 1/33 to 30 seconds. The 1200 second dark had a mean K value of 3 ADU.

I believe that the final answer is "Pentax is messing with the data before RAW" which isn't surprising.

Thanks for your effort and sharing results!
I use the following dcraw parameters: -d -r 1 1 1 1 -4 -k 0 -S 65535. The use of -k 0 ("Set the darkness level") is important because it tells dcraw not to use the value provided with the corresponding exif tag. Of cause, there is no difference if the exif tag value is 0 (as in manual exposures >10 seconds and bulb's on my cam). On the other hand I do not know if your version of dcraw has the very same parameters as mine.

---

Last edited by ewelot; 03-09-2013 at 01:44 PM. Reason: EDIT: changed dcraw parameter (originally I wrote "-S 16383", which was not correct)

[smigol]

I'm not alone anymore ...! (I started using Linux in 1992 - just for the courious.)



Thanks for your effort and sharing results!
I use the following dcraw parameters: -d -r 1 1 1 1 -4 -k 0 -S 16383. The use of -k 0 ("Set the darkness level") is important because it tells dcraw not to use the value provided with the corresponding exif tag. Of cause, there is no difference if the exif tag value is 0 (as in manual exposures >10 seconds and bulb's on my cam). On the other hand I do not know if your version of dcraw has the very same parameters as mine.

I ran my 1200 sec sub through dcraw with another setting that the man page suggests to create a dark frame: dcraw -D -T -4 -j -t 0
This is the source of the picture on the left, below.
Note the values reported by Maxim. You can also see some of the hot pixels visible and the amp glow. The hump in the histogram is at about 120 ADU.
I reran dcraw using your settings you list above for comparison. The -d option will scale and thus the snip on the right below shows a different ADU average. The hump in this example is about 450 ADU.

When I compare subs across a range of temperatures from 0C to this one at 29C, the hump migrates out of the 0 spike. There is no sudden jump.

I'll try to check against a selection of other files I have around at different ISO settings.

I hope that with the right work arounds the K5IIs will be a good camera for your efforts. Are you planning to remove the hot filter?

[sbruzzone]

I ran my 1200 sec sub through dcraw with another setting that the man page suggests to create a dark frame: dcraw -D -T -4 -j -t 0
This is the source of the picture on the left, below.
Note the values reported by Maxim. You can also see some of the hot pixels visible and the amp glow. The hump in the histogram is at about 120 ADU.
I reran dcraw using your settings you list above for comparison. The -d option will scale and thus the snip on the right below shows a different ADU average. The hump in this example is about 450 ADU.

When I compare subs across a range of temperatures from 0C to this one at 29C, the hump migrates out of the 0 spike. There is no sudden jump.

I'll try to check against a selection of other files I have around at different ISO settings.

I hope that with the right work arounds the K5IIs will be a good camera for your efforts. Are you planning to remove the hot filter?

Hey guys! I've just joined the forum because I'm planning to get a K5IIs also for astro pictures. I've found this topic very interesting, and I'm wondering what's going on with those dark files.
I have experience processing images and It looks that some date is truncated or altered to get that histograms. Of course it should be gaussian, any numbers about the RDNOISE or GAIN for this camera. Have you tried IRAF on those images? The temp shouldn't be a problem here and the glow seem reasonable. No experience with this cameras, yet (
I don't know the ADUs added by the camera to get rid of the negative scores. (300?).
Cheers -Sebastian

[smigol]

Welcome Sebastian,

Be careful to note that Thomas is the one with the K5IIs, the camera data that I'm showing is from a very old K10D.

From what we've been able to see, there is some truncating going on, much like what Canon does with their cameras. The question remains what thresholds are the triggers - Thomas has found one at 10 seconds as manual vs bulb. From testing on the Canon methods, there should be investigation on ISO and temperature as triggers for data truncation.

I am also curious about what can be done with this camera because it might be my next astrocamera, too.

Stephen

[ewelot]

Stephen,

sorry, I did a mistake: the dcraw parameters I posted above are used in my astro image reduction. I do scale up couts from the 14bit to 16bit range. All my dark measurements discussed here were done at 14bit and therefore use the "-D" switch (same result with "-d -S 65535"). Thanks a lot for your numbers again which helped to discover this error. I apologize for that!

Welcome Sebastian,

Be careful to note that Thomas is the one with the K5IIs, the camera data that I'm showing is from a very old K10D.

From what we've been able to see, there is some truncating going on, much like what Canon does with their cameras. The question remains what thresholds are the triggers - Thomas has found one at 10 seconds as manual vs bulb. From testing on the Canon methods, there should be investigation on ISO and temperature as triggers for data truncation.

I am also curious about what can be done with this camera because it might be my next astrocamera, too.

Stephen

The big difference between Canon's and Pentax's hidden in camera pre-processing is that Canon does not move the black point level to zero and therefore does NOT truncate data! They do apply a shift - probably to compensate for dark current (as noted by Craig Stark). You would not be able to measure the dark count anymore but that is nothing to worry about. The main point is that the dark pattern and noise is kept intact and can perfectly be used in the image processing pipeline.

Still I have some hope on using the K5IIs in astrophotography. I'll certainly try hard to get as much out of it as possible but the blackpoint issue will probably not allow to take advantage of full performance of the sensor.

[smigol]

Stephen,

sorry, I did a mistake: the dcraw parameters I posted above are used in my astro image reduction. I do scale up couts from the 14bit to 16bit range. All my dark measurements discussed here were done at 14bit and therefore use the "-D" switch (same result with "-d -S 65535"). Thanks a lot for your numbers again which helped to discover this error. I apologize for that!



The big difference between Canon's and Pentax's hidden in camera pre-processing is that Canon does not move the black point level to zero and therefore does NOT truncate data! They do apply a shift - probably to compensate for dark current (as noted by Craig Stark). You would not be able to measure the dark count anymore but that is nothing to worry about. The main point is that the dark pattern and noise is kept intact and can perfectly be used in the image processing pipeline.

Still I have some hope on using the K5IIs in astrophotography. I'll certainly try hard to get as much out of it as possible but the blackpoint issue will probably not allow to take advantage of full performance of the sensor.

Thomas, I remember that you said you were testing at ISO 800. Are you seeing similar patterns at ISO 100 (or does the K5IIs go to 80)? I wonder if this truncation is worsened by this setting.

[TrueFocus]

I hope I haven't missed something among the way, but have you checked the "Slow Shutter Speed NR" option? It performs a dark frame substraction in-camera for long exposures, due to the nature of the process it would have to apply to RAW files as well.

[ewelot]

Thomas, I remember that you said you were testing at ISO 800. Are you seeing similar patterns at ISO 100 (or does the K5IIs go to 80)? I wonder if this truncation is worsened by this setting.

Stephen, I just did some little testing at ISO 100 (dark room, near 20C), with similar results but somewhat different numbers:
- all exposures in bulb mode (even at shortest exposure times) have blackpoint = 0, the lower half of the gaussian noise distribution is truncated to zero (about 70% of all pixels are at 0 level using the same sensor area as in my earlier histogram measurements)
- all exposures in manual mode up until 30 seconds have blackpoint near 67
So the difference is in manual mode: blackpoint is near 512/8 and even longer exposures (30 seconds) do not suffer from the "blackpoint issue" (data clipping).

I hope I haven't missed something among the way, but have you checked the "Slow Shutter Speed NR" option? It performs a dark frame substraction in-camera for long exposures, due to the nature of the process it would have to apply to RAW files as well.

Slow shutter speed NR has been disabled.

[TrueFocus]

If I remember correctly there were additional NR settings in the debug menu.
If PK-Tether works with the K5iis then you could try accessing it.