Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
03-08-2013, 10:02 AM   #16
Site Supporter
smigol's Avatar

Join Date: Sep 2008
Location: Menlo Park, CA
Photos: Albums
Posts: 402
Thomas,

I did some testing of a K10D yesterday to see what happens with the black point and histogram at the 1, 5, 10, 15, and 30 second durations with manual and bulb exposures. No strict temperature controls applied at the time, it was just at ambient and didn't make the camera reach thermal equilibrium. Still, the temperatures are mostly stable because the durations were short. At 100 ISO, the blackpoints are all 0. The mean is uniformly .7 to .8 ADU. Max was due to hot pixels and min was due to dead pixels.

When stretching the histogram to show the full range, it looks more like your second example - a truncated curve with a tail leading off to the left with a spike.


The background bias noise in the above example has been stretched to show the histogram. The histogram is generated in the above view by averaging the 3x3 area around the mouse pointer which is placed at the center of the image.



This view above is the same sub with a 300% view. This shows the density of the bias noise. Note the location of the hot pixels as large blocks of color. Also note the lack of black pixels - maybe this is a view of properly non-truncated data?

I do see bias in the darks Thomas has posted (and the ones I generated of short duration), so it's the prominent source of noise as the vertical pattern suggests. The ring of glow around the edges in the K5IIs example looks like classic amp glow and matches what I saw in a dark that I've seen on a K5.

To answer the question of why did the engineers do this, I'd point to what Craig Stark says in his conclusion in the Canon study. It's to optimize the performance of the camera for the common daylight use. There is some tradeoff of dark signal being lost, maybe it's not a full stop like Stark found, but it is a sacrifice. The question is still raised, why is it not scaled under common exposures (less than 10 seconds) and then turned on by default at anything above? Thomas is right, this seems to be a reverse of what would be desired. Again, my guess is that the engineers may be thinking less about saving the dim data and more about saving headroom for highlights.

What's the remedy? A firmware update would be nice that gives a toggle to reset black point - that would be ideal. Maybe Pentax will give us this control if they did, bravo, if not, we may have to use workarounds of tricking the camera and standardizing on this behavior.

In practice, it seems that if the camera's performance is profiled at a particular exposure duration and ISO setting, a map of noise across a range of temperatures would generate a usable reduction master. As these cameras are not temperature regulated, we have to depend on the EXIF temp reading and matching darks to lights to develop a profile. In my experience, this linking of temperatures needs to be within 2 C to be usable. Anything more does not scale properly.

Once we decide on the setting that works for the conditions used (for me, I've said it's 1200 second subs at 100 ISO) then all the rest of the efforts revolve around that point. Mount tuning, alignment, library management, and power supply are all part of the effort to support this choice. Maybe these long durations are enough to get the heat signal out of the murk. Maybe not as even a 1200 second 29C sub I referenced had a black point of 0 and mean of 3 (median of 0).

03-08-2013, 10:17 AM   #17
Site Supporter




Join Date: Mar 2008
Location: Prince George, BC
Photos: Gallery
Posts: 2,322
As an aside to one of your points, I have found that the time and temperature of a long exposure as written to the EXIF is that of the start of the exposure not the end. Have you also found that?

Jack
03-08-2013, 10:39 AM   #18
Site Supporter
smigol's Avatar

Join Date: Sep 2008
Location: Menlo Park, CA
Photos: Albums
Posts: 402
I see that, too. The EXIF date taken value is the start and the file system "date" value is the time it was written. Easy to see with long exposures.
The temp data seems to be the same as well. Easy to test, too. Refrigerate the camera, start a long exposure, put it in a warm environment so that during the exposure it heats up. The EXIF temp will be the cold value. Also, it's well known that the EXIF temp is from a sensor that is not close to the image sensor. It's someplace else and is only good for general use.
03-08-2013, 11:57 AM   #19
New Member




Join Date: Mar 2013
Location: Weimar, Germany
Photos: Albums
Posts: 22
Original Poster
Stephen,

just a small note about your measurements. From the images you posted I assume you measured the dark in RGB images. I'd suggest you do this in the gray data before they are de-bayered. Otherwise - regardless of the algorithm used for de-bayering - data are interpolated which heavily affects the histogram (e.g. the number of pixels found at ADU=0 decreases a lot due to de-bayer algorithm).
The same applies for dark image reduction: you should always subtract dark from light exposures before de-bayering. But maybe you did it already this way.

03-08-2013, 12:32 PM   #20
Pentaxian




Join Date: Jul 2009
Location: Seattle
Posts: 7,089
QuoteOriginally posted by ewelot Quote
...
but I run a Linux OS which is probably somewhat exotic amongst users of this forum.
There are more using it than you might think.

My first installation of Linux was around 1994. I had it for a main desktop at one point when it got better but eventually the I had to get a different UNIX system that supported secret, proprietary file formats for video editing and could run more popular image editors that supported more RAW files and could handle 48 bit color. People using Linux today have it way-easy compared to back then.
03-08-2013, 12:33 PM   #21
Site Supporter
smigol's Avatar

Join Date: Sep 2008
Location: Menlo Park, CA
Photos: Albums
Posts: 402
Yes, I do all calibration before debayering. The screengrabs were from fastpictureviewer as it was quick and easy. I did my testing yesterday with dcraw generated tiffs with -D -r 1 1 1 1 -4 -T. Then I fed these into PixInsight to get the mean and median values.

K values ranged from .7 to .9 ADU for durations from 1/33 to 30 seconds. The 1200 second dark had a mean K value of 3 ADU.

I believe that the final answer is "Pentax is messing with the data before RAW" which isn't surprising.

Last edited by smigol; 03-08-2013 at 12:45 PM.
03-08-2013, 02:16 PM   #22
New Member




Join Date: Mar 2013
Location: Weimar, Germany
Photos: Albums
Posts: 22
Original Poster
QuoteOriginally posted by tuco Quote
There are more using it than you might think.

My first installation of Linux was around 1994. I had it for a main desktop at one point when it got better but eventually the I had to get a different UNIX system that supported secret, proprietary file formats for video editing and could run more popular image editors that supported more RAW files and could handle 48 bit color. People using Linux today have it way-easy compared to back then.
I'm not alone anymore ...! (I started using Linux in 1992 - just for the courious.)

QuoteOriginally posted by smigol Quote
Yes, I do all calibration before debayering. The screengrabs were from fastpictureviewer as it was quick and easy. I did my testing yesterday with dcraw generated tiffs with -D -r 1 1 1 1 -4 -T. Then I fed these into PixInsight to get the mean and median values.

K values ranged from .7 to .9 ADU for durations from 1/33 to 30 seconds. The 1200 second dark had a mean K value of 3 ADU.

I believe that the final answer is "Pentax is messing with the data before RAW" which isn't surprising.
Thanks for your effort and sharing results!
I use the following dcraw parameters: -d -r 1 1 1 1 -4 -k 0 -S 65535. The use of -k 0 ("Set the darkness level") is important because it tells dcraw not to use the value provided with the corresponding exif tag. Of cause, there is no difference if the exif tag value is 0 (as in manual exposures >10 seconds and bulb's on my cam). On the other hand I do not know if your version of dcraw has the very same parameters as mine.

Last edited by ewelot; 03-09-2013 at 01:44 PM. Reason: EDIT: changed dcraw parameter (originally I wrote "-S 16383", which was not correct)
03-08-2013, 03:08 PM   #23
Site Supporter
smigol's Avatar

Join Date: Sep 2008
Location: Menlo Park, CA
Photos: Albums
Posts: 402
QuoteOriginally posted by ewelot Quote
I'm not alone anymore ...! (I started using Linux in 1992 - just for the courious.)



Thanks for your effort and sharing results!
I use the following dcraw parameters: -d -r 1 1 1 1 -4 -k 0 -S 16383. The use of -k 0 ("Set the darkness level") is important because it tells dcraw not to use the value provided with the corresponding exif tag. Of cause, there is no difference if the exif tag value is 0 (as in manual exposures >10 seconds and bulb's on my cam). On the other hand I do not know if your version of dcraw has the very same parameters as mine.
I ran my 1200 sec sub through dcraw with another setting that the man page suggests to create a dark frame: dcraw -D -T -4 -j -t 0
This is the source of the picture on the left, below.
Note the values reported by Maxim. You can also see some of the hot pixels visible and the amp glow. The hump in the histogram is at about 120 ADU.
I reran dcraw using your settings you list above for comparison. The -d option will scale and thus the snip on the right below shows a different ADU average. The hump in this example is about 450 ADU.

When I compare subs across a range of temperatures from 0C to this one at 29C, the hump migrates out of the 0 spike. There is no sudden jump.

I'll try to check against a selection of other files I have around at different ISO settings.

I hope that with the right work arounds the K5IIs will be a good camera for your efforts. Are you planning to remove the hot filter?

Attached Images
   

Last edited by smigol; 03-08-2013 at 03:16 PM.
03-08-2013, 10:08 PM   #24
Forum Member




Join Date: Feb 2013
Location: London, ON
Photos: Albums
Posts: 64
QuoteOriginally posted by smigol Quote
I ran my 1200 sec sub through dcraw with another setting that the man page suggests to create a dark frame: dcraw -D -T -4 -j -t 0
This is the source of the picture on the left, below.
Note the values reported by Maxim. You can also see some of the hot pixels visible and the amp glow. The hump in the histogram is at about 120 ADU.
I reran dcraw using your settings you list above for comparison. The -d option will scale and thus the snip on the right below shows a different ADU average. The hump in this example is about 450 ADU.

When I compare subs across a range of temperatures from 0C to this one at 29C, the hump migrates out of the 0 spike. There is no sudden jump.

I'll try to check against a selection of other files I have around at different ISO settings.

I hope that with the right work arounds the K5IIs will be a good camera for your efforts. Are you planning to remove the hot filter?
Hey guys! I've just joined the forum because I'm planning to get a K5IIs also for astro pictures. I've found this topic very interesting, and I'm wondering what's going on with those dark files.
I have experience processing images and It looks that some date is truncated or altered to get that histograms. Of course it should be gaussian, any numbers about the RDNOISE or GAIN for this camera. Have you tried IRAF on those images? The temp shouldn't be a problem here and the glow seem reasonable. No experience with this cameras, yet (
I don't know the ADUs added by the camera to get rid of the negative scores. (300?).
Cheers -Sebastian
03-09-2013, 12:00 AM   #25
Site Supporter
smigol's Avatar

Join Date: Sep 2008
Location: Menlo Park, CA
Photos: Albums
Posts: 402
Welcome Sebastian,

Be careful to note that Thomas is the one with the K5IIs, the camera data that I'm showing is from a very old K10D.

From what we've been able to see, there is some truncating going on, much like what Canon does with their cameras. The question remains what thresholds are the triggers - Thomas has found one at 10 seconds as manual vs bulb. From testing on the Canon methods, there should be investigation on ISO and temperature as triggers for data truncation.

I am also curious about what can be done with this camera because it might be my next astrocamera, too.

Stephen
03-09-2013, 01:39 PM   #26
New Member




Join Date: Mar 2013
Location: Weimar, Germany
Photos: Albums
Posts: 22
Original Poster
Stephen,

sorry, I did a mistake: the dcraw parameters I posted above are used in my astro image reduction. I do scale up couts from the 14bit to 16bit range. All my dark measurements discussed here were done at 14bit and therefore use the "-D" switch (same result with "-d -S 65535"). Thanks a lot for your numbers again which helped to discover this error. I apologize for that!

QuoteOriginally posted by smigol Quote
Welcome Sebastian,

Be careful to note that Thomas is the one with the K5IIs, the camera data that I'm showing is from a very old K10D.

From what we've been able to see, there is some truncating going on, much like what Canon does with their cameras. The question remains what thresholds are the triggers - Thomas has found one at 10 seconds as manual vs bulb. From testing on the Canon methods, there should be investigation on ISO and temperature as triggers for data truncation.

I am also curious about what can be done with this camera because it might be my next astrocamera, too.

Stephen
The big difference between Canon's and Pentax's hidden in camera pre-processing is that Canon does not move the black point level to zero and therefore does NOT truncate data! They do apply a shift - probably to compensate for dark current (as noted by Craig Stark). You would not be able to measure the dark count anymore but that is nothing to worry about. The main point is that the dark pattern and noise is kept intact and can perfectly be used in the image processing pipeline.

Still I have some hope on using the K5IIs in astrophotography. I'll certainly try hard to get as much out of it as possible but the blackpoint issue will probably not allow to take advantage of full performance of the sensor.
03-10-2013, 01:28 PM   #27
Site Supporter
smigol's Avatar

Join Date: Sep 2008
Location: Menlo Park, CA
Photos: Albums
Posts: 402
QuoteOriginally posted by ewelot Quote
Stephen,

sorry, I did a mistake: the dcraw parameters I posted above are used in my astro image reduction. I do scale up couts from the 14bit to 16bit range. All my dark measurements discussed here were done at 14bit and therefore use the "-D" switch (same result with "-d -S 65535"). Thanks a lot for your numbers again which helped to discover this error. I apologize for that!



The big difference between Canon's and Pentax's hidden in camera pre-processing is that Canon does not move the black point level to zero and therefore does NOT truncate data! They do apply a shift - probably to compensate for dark current (as noted by Craig Stark). You would not be able to measure the dark count anymore but that is nothing to worry about. The main point is that the dark pattern and noise is kept intact and can perfectly be used in the image processing pipeline.

Still I have some hope on using the K5IIs in astrophotography. I'll certainly try hard to get as much out of it as possible but the blackpoint issue will probably not allow to take advantage of full performance of the sensor.
Thomas, I remember that you said you were testing at ISO 800. Are you seeing similar patterns at ISO 100 (or does the K5IIs go to 80)? I wonder if this truncation is worsened by this setting.
03-11-2013, 01:41 AM   #28
Forum Member




Join Date: Apr 2012
Posts: 78
I hope I haven't missed something among the way, but have you checked the "Slow Shutter Speed NR" option? It performs a dark frame substraction in-camera for long exposures, due to the nature of the process it would have to apply to RAW files as well.
03-11-2013, 02:08 PM   #29
New Member




Join Date: Mar 2013
Location: Weimar, Germany
Photos: Albums
Posts: 22
Original Poster
QuoteOriginally posted by smigol Quote
Thomas, I remember that you said you were testing at ISO 800. Are you seeing similar patterns at ISO 100 (or does the K5IIs go to 80)? I wonder if this truncation is worsened by this setting.
Stephen, I just did some little testing at ISO 100 (dark room, near 20C), with similar results but somewhat different numbers:
- all exposures in bulb mode (even at shortest exposure times) have blackpoint = 0, the lower half of the gaussian noise distribution is truncated to zero (about 70% of all pixels are at 0 level using the same sensor area as in my earlier histogram measurements)
- all exposures in manual mode up until 30 seconds have blackpoint near 67
So the difference is in manual mode: blackpoint is near 512/8 and even longer exposures (30 seconds) do not suffer from the "blackpoint issue" (data clipping).

QuoteOriginally posted by TrueFocus Quote
I hope I haven't missed something among the way, but have you checked the "Slow Shutter Speed NR" option? It performs a dark frame substraction in-camera for long exposures, due to the nature of the process it would have to apply to RAW files as well.
Slow shutter speed NR has been disabled.
03-11-2013, 02:29 PM   #30
Forum Member




Join Date: Apr 2012
Posts: 78
If I remember correctly there were additional NR settings in the debug menu.
If PK-Tether works with the K5iis then you could try accessing it.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
bulb, camera, dslr, exposure, exposures, images, k-5, k-5 ii, k-5 iis, k5, mode, pentax k-5, time
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
Is Histogram RAW or JPG? w1wbz Visitors' Center 5 07-01-2013 08:52 AM
K5 RAW Preview Image is Desaturated/Overexposed, RAW Correct. WTF? Epoch of Entropy Pentax K-5 15 11-15-2012 10:47 AM
PDCU: overwrite RAW or save altered RAW-file Alfie Digital Processing, Software, and Printing 4 09-27-2012 09:33 PM
why not in raw? bull drinkwater Pentax Medium Format 4 06-20-2012 04:28 PM
Why do RAW my RAW images not look great... pjthiel Digital Processing, Software, and Printing 23 05-17-2011 07:00 AM



All times are GMT -7. The time now is 09:01 AM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top