Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
12-10-2018, 10:40 PM   #1
New Member




Join Date: Dec 2018
Posts: 14
Black Level and Black Point issue

Hello all

I'm having trouble with DeepSkyStacker, which use DCRaw to load raw files. Some files appear "redish".

Name:  post-301818-0-25891600-1541119062.jpg
Views: 501
Size:  41.9 KB
Name:  post-301818-0-85675100-1541119068.jpg
Views: 540
Size:  43.2 KB

The issue relies with the black point (full story here), and I found that my Pentax K5-II set EXIF "Black Point" and "Black Level" tags differently depending on the mode and ISO level :

When in Bulb mode, regardless of ISO :
Code:
Black Point                     : 0 0 0 0
Black Level : 0 0 0 0
When not in Bulb mode, ISO =< 800, something like :
Code:
Black Point                     : 513 513 515 515
Black Level : 513 513 515 515
When not in Bulb mode, ISO > 800 :
Code:
Black Point                     : 0 0 0 0
Black Level : 0 0 0 0
Does anyone have the slightest idea what's happening here ? Is this a camera firmware bug ?
Some test DNG can be downloaded here.

Thanks !

12-10-2018, 11:20 PM   #2
Otis Memorial Pentaxian
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 42,007
QuoteOriginally posted by blemasle Quote
I found that my Pentax K5-II set EXIF "Black Point" and "Black Level" tags differently depending on the mode and ISO level :
I take it you are the same blemasle that started the thread at DeepSkyStacker

I have downloaded the test DNG files. Do you have a key for what each file is?

I am not sure that EXIF is much help in that the settings it records are often not related to how the A/D convertor and low level processor does its work in assigning 14-bit values set in the RAW data. That being said, conventional wisdom has been (based on anecdotal accounts) that it is a good idea to disable High ISO NR and Slow Shutter Speed NR as well as D-Range Highlight and Shadow Correction for technical work. Any one of the above might show the EXIF pattern you are seeing as a side effect.

I took a look at the test DNG and so far, I am not seeing the pink rendering with the in-camera JPEG previews, but will process in dcraw and see what comes out.


Steve
12-10-2018, 11:25 PM   #3
Otis Memorial Pentaxian
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 42,007
BTW...what EXIF parser are you using to extract "Black Point" and "Black Level"?

Also, do you have verbose output from dcraw for its actions as well as the flags passed to dcraw for processing?


Steve
12-11-2018, 12:01 AM   #4
New Member




Join Date: Dec 2018
Posts: 14
Original Poster
Thanks for helping me out !
All those options were disabled when those DNG were shot. But Pentax (or at least my K5-II) is known for having some forced noise reduction for ISO 1600 and up, right ?

I'm using exiftool.
Below are the extracted relevant value for each file in the zip. The files with 0 Black Point/Level are the one that seems to work correctly with DeepSkyStacker.

Code:
======== IMGP9834.DNG
Black Point : 67 67 67 67
Black Level : 67 67 67 67
Picture Mode : Manual, Off-Auto-Aperture; 1/3 EV steps
Exposure Time : 1/8000
ISO : 100
======== IMGP9835.DNG
Black Point : 129 129 130 129
Black Level : 129 129 130 129
Picture Mode : Manual, Off-Auto-Aperture; 1/3 EV steps
Exposure Time : 1/8000
ISO : 200
======== IMGP9836.DNG
Black Point : 257 257 258 258
Black Level : 257 257 258 258
Picture Mode : Manual, Off-Auto-Aperture; 1/3 EV steps
Exposure Time : 1/8000
ISO : 400
======== IMGP9837.DNG
Black Point : 513 513 515 515
Black Level : 513 513 515 515
Picture Mode : Manual, Off-Auto-Aperture; 1/3 EV steps
Exposure Time : 1/8000
ISO : 800
======== IMGP9838.DNG
Black Point : 0 0 0 0
Black Level : 0 0 0 0
Picture Mode : Manual, Off-Auto-Aperture; 1/3 EV steps
Exposure Time : 1/8000
ISO : 1600
======== IMGP9839.DNG
Black Point : 0 0 0 0
Black Level : 0 0 0 0
Picture Mode : Manual, Off-Auto-Aperture; 1/3 EV steps
Exposure Time : 1/8000
ISO : 3200
======== IMGP9922.DNG
Black Point : 511 511 513 512
Black Level : 511 511 513 512
Picture Mode : Manual, Off-Auto-Aperture; 1/3 EV steps
Exposure Time : 2
ISO : 800
======== IMGP9924.DNG
Black Point : 0 0 0 0
Black Level : 0 0 0 0
Picture Mode : Bulb, Off-Auto-Aperture; 1/3 EV steps
Exposure Time : 1.9
ISO : 800
======== IMGP9926.DNG
Black Point : 0 0 0 0
Black Level : 0 0 0 0
Picture Mode : Manual, Off-Auto-Aperture; 1/3 EV steps
Exposure Time : 2
ISO : 1600
======== IMGP9927.DNG
Black Point : 0 0 0 0
Black Level : 0 0 0 0
Picture Mode : Bulb, Off-Auto-Aperture; 1/3 EV steps
Exposure Time : 1.9
ISO : 1600
Let me know if I can be of any further help !

12-11-2018, 12:14 AM   #5
New Member




Join Date: Dec 2018
Posts: 14
Original Poster
Sorry, I forgot about that part. I do not have the flags used by DeepSkyStacker to load the raws. I'll try to look if I can find something about that, as I just found out that DSS is now open source and hosted on github.
12-11-2018, 12:35 AM   #6
New Member




Join Date: Dec 2018
Posts: 14
Original Poster
From what I can tell from the source code, DSS is using the following command to load my DNG files.

Code:
dcraw.exe  -j -k 0 -t 0 -4 -T {file}
The "-T" flag is just to output tiff files.
The "-k 0" set the black point to 0. If set on "faulty" files, I do get a redish tiff as an output.
12-11-2018, 12:49 AM - 1 Like   #7
Otis Memorial Pentaxian
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 42,007
Thanks for the replies. The dcraw command is the smoking gun.

While your were posting, I was able to reproduce the pink dark-subtract frame by either allowing dcraw to set its own black point or by setting the black point to 0 (essentially the same since all values are near zero). This happens for both M mode and B mode exposures at both low and high ISOs. Configuring dcraw to make a so-called linear TIFF (no curves applied) as recommended for dark-frame subtraction provides the expected raw pixel to output pixel mapping for the subtract all dark.

I concur with the advice on the CloudyNights site to uncheck "Set black point to 0" and believe that the pink is a processing side-effect of misdirecting dcraw. I also strongly suspect that the EXIF entries for black point are a red herring and reflect settings used for the in-camera preview JPEG creation.

Directives for back-ground subtract image.
Code:
dcraw -v -D -4 -j -t 0 -T <filename>

Steve


Last edited by stevebrot; 12-11-2018 at 01:02 AM.
12-11-2018, 01:00 AM   #8
Otis Memorial Pentaxian
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 42,007
QuoteOriginally posted by stevebrot Quote
While your were posting, I was able to reproduce the pink dark-subtract frame by either allowing dcraw to set its own black point or by setting the black point to 0 (essentially the same since all values are near zero).
Default settings to produce a 16-bit TIFF with the camera-designating white balance.
Code:
dcraw -v -6 -T -w <filename>
--or--

The same with black point set to 0
Code:
dcraw -v -k 0 -6 -T -w <filename>
Both create pink TIFFs for all of your test files.


Steve
12-11-2018, 02:08 AM   #9
New Member




Join Date: Dec 2018
Posts: 14
Original Poster
Thanks a lot for the time you're investing into this !

QuoteOriginally posted by stevebrot Quote
Default settings to produce a 16-bit TIFF with the camera-designating white balance.
Code:
dcraw -v -6 -T -w <filename>
The default settings of DSS is -4 (not -6) which according to the help is equivalent to "-6 -W -g 1 1". The important part seems to be the "-W". Without it, I can indeed get redish tint on all my images. With it (like DSS), I reproduce my issue on the file with a Black Point set to anything else than 0.

QuoteOriginally posted by stevebrot Quote
I also strongly suspect that the EXIF entries for black point are a red herring and reflect settings used for the in-camera preview JPEG creation
The Black Point EXIF seems to be used by dcraw :

Code:
>dcraw -v -j -t 0 -4 -d -T IMGP9837.DNG

Loading Pentax K-5 II image from IMGP9837.DNG ...
Scaling with darkness 513
And when "-k 0" is applied it gets discarded, which result in a redish image :
Code:
>dcraw -v -j -t 0 -4 -d -T IMGP9837.DNG

Loading Pentax K-5 II image from IMGP9837.DNG ...
Scaling with darkness 0
Finally, If I edit the EXIF Black Point/Level to 0 0 0 0, both command lines results in a redish tint. Setting the black point to its original value using "-k 513" produce a black image.
Code:
exiftool "-BlackPoint=0 0 0 0" "-BlackLevel=0 0 0 0" "IMGP9837 - Copy.DNG"
So the EXIF tag is vital for dcraw to load the image correctly.

From all that, I should indeed uncheck "Set black point to 0" at all times
I'm puzzled why the camera does change those EXIF and apparently the image data itself (if I'm getting this right) depending on the mode or ISO setting though ? Am I missing something from your explanation ?
12-11-2018, 09:15 AM   #10
Otis Memorial Pentaxian
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 42,007
QuoteOriginally posted by blemasle Quote
The Black Point EXIF seems to be used by dcraw :
Interesting. I guess that is why Dave Coffin suggests the -D option to avoid scaling (rather than -d). I might suggest this behavior might want to be referred to the authors of DSS.

Manpage of dcraw (Note on -K option)


Steve

Last edited by stevebrot; 12-11-2018 at 11:21 AM.
12-11-2018, 12:35 PM   #11
Otis Memorial Pentaxian
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 42,007
QuoteOriginally posted by blemasle Quote
I'm puzzled why the camera does change those EXIF and apparently the image data itself (if I'm getting this right) depending on the mode or ISO setting though ? Am I missing something from your explanation ?
Nope, you are not missing anything.

What I have not been able to determine is the source of the darkness setting used by dcraw. Standard EXIF has a BlackLevel tag that reports four values taken from four actual pixels in the cropped out upper left of the image and those vary, apparently according ISO and mode. Pentax makernotes has a BlackPoint tag with the same four values, presumably derived from same source. Remember that the EXIF (makernotes included) are descriptive and not directive, though there may be clues that may be leveraged by processing tools downstream.

FWIW, dcraw appears to use the value from EXIF:BlackLevel for its messaging, but not for the actual processing when using "dcraw -v -j -t 0 -4 -d -T".


Steve
12-11-2018, 04:07 PM   #12
New Member




Join Date: Dec 2018
Posts: 14
Original Poster
Maybe just a confusion between -d and -D ?
I'll try to reach the maintainers of the Gihub repo.

We still miss an explanation of why the data is different depending on mode or ISO though.

I'll try to update this thread if anything comes up with DSS maintainers

Thanks again !
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
bulb, files, iso, issue, level, mode, photography, photoshop
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
Showstopper level issue with star trail stacking MJKoski Pentax KP 7 02-21-2018 10:17 AM
Customer vs. Nikon (D500 issue) - taking it to the next level beholder3 Photographic Industry and Professionals 2 06-16-2016 01:19 AM
How to improve K-3 DA* shots to contemporary Point&Shoot level? Prakticant Pentax K-3 & K-3 II 14 02-14-2015 09:15 AM
An Entry Level Point-n-Shoot of the Future ? adr1an Canon, Nikon, Sony, and Other Camera Brands 6 08-14-2011 08:19 AM
k-5 issue with the electronic level :( bsimon Pentax K-5 & K-5 II 20 10-29-2010 08:13 PM



All times are GMT -7. The time now is 06:10 AM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top