[kh1234567890]

RAW files have no histograms per se.

Of course they do. Any dataset can be plotted as a histogram. After demosaicing and gamma mapping the actual bin counts will be different but hard clipped data will never come back. The LCD histogram is largely a useless gimmick, turning on the highlight correction and letting the camera do the work is fine 99% of the time.

[BrianR]

Of course they do. Any dataset can be plotted as a histogram.

And some info on RawDigger's Raw histogram for the interested:

RawDigger histograms: Part 1. What is the raw data histogram? | RawDigger

[ScooterMaxi Jim]

The K-3 is clearly a more sophisticated metering system than the previous Pentax implementations. What I have found, unfortunately, is it proves challenging to create a single recipe to ensure consistent exposures. In extreme contrast conditions, I will tend to set EV to -0.7 AND have Auto setting for Highlight Compensation. On the K-3 in Auto HC, the highlight compensation will only kick in if significant blowout is occurring - so there is 0 down side to leaving Auto HC on.

As for how the RAW preserves highlights beyond what the JPEG records, every scene will be different - and I'm finding considerable variability based on the particular lens in use. As a general rule, I can live with a bit of underexposure for safety's sake. I wish the K-3 was calibrated more closely to exposure values found on the K-30. I know a lot of folks view the typical profile as underexposure, but I am much more comfortable with safe (under) rather than sorry (lost data).

[stevebrot]

Of course they do. Any dataset can be plotted as a histogram.

That is why I said per se. There is a frequency diagram* (histogram) that could be plotted, but the chart would not be as useful as one based on an actual constructed image. The article linked by BrianR very nicely illustrates why the RAW histogram is something we never see when using our cameras or common PP tools.

And some info on RawDigger's Raw histogram for the interested:

RawDigger histograms: Part 1. What is the raw data histogram? | RawDigger

Steve

* On a histogram, the Y axis is always the number of occurrences (frequency) with the total of each column being the total number of counts (in this case pixels). Histograms are not the same as bar graphs or response curves.

---

Last edited by stevebrot; 08-16-2015 at 12:17 PM.

[BrianR]

There is a frequency diagram* (histogram) that could be plotted, but the chart would not be as useful as one based on an actual constructed image.

I don't agree with that, I'd say it's differently useful. The image constructed by the cameras jpeg engine won't be the same as the image I'll construct with my raw converter, so I'm basically stuck with trying to interpret how the in-camera jpeg reflects the captured raw data. If you stick with one set of relatively neutral jpeg settings over time you can get pretty good at this, but I'd prefer the option for the camera to show the raw data.

[IchabodCrane]

I don't agree with that, I'd say it's differently useful. The image constructed by the cameras jpeg engine won't be the same as the image I'll construct with my raw converter, so I'm basically stuck with trying to interpret how the in-camera jpeg reflects the captured raw data. If you stick with one set of relatively neutral jpeg settings over time you can get pretty good at this, but I'd prefer the option for the camera to show the raw data.

Is the histogram displayed by the RAW converter strictly of the data itself or is it based on how the software interprets the data into a displayed image (is that a JPEG or a TIFF?) which might be different than the camera's interpretation? In other words, is there really such a thing as a histogram of RAW data?

[ScooterMaxi Jim]

Is the histogram displayed by the RAW converter strictly of the data itself or is it based on how the software interprets the data into a displayed image (is that a JPEG or a TIFF?) which might be different than the camera's interpretation? In other words, is there really such a thing as a histogram of RAW data?

At least on a good RAW converter, it is a simple charting of the RAW data - most often altered based on the marker settings (i.e. Natural or Bright, etc.) provided for conversion. All converters need some sort of profile to translate the data into a histogram that tells you expected output; but the RAW data remains whole. The difference in the converter vs. what is seen on the LCD screen is related to the lost data in providing an actual JPEG thumbnail.

[stevebrot]

Is the histogram displayed by the RAW converter strictly of the data itself or is it based on how the software interprets the data into a displayed image (is that a JPEG or a TIFF?) which might be different than the camera's interpretation? In other words, is there really such a thing as a histogram of RAW data?

It depends on the RAW converter. Some will use the embedded JPEG in the RAW file as the basis for the displayed histogram. Others (e.g. Lightroom) generate a new JPEG thumbnail using the converter's import settings and use that to generate the displayed histogram. I don't know of any RAW converter that displays a histogram derived directly from the capture data.

In other words, is there really such a thing as a histogram of RAW data?

As noted in the comments above, you can construct a histogram directly from the data in the PEF/DNG file. The article posted by BrianR covers the subject in detail and provides a good discussion of how to use the information. For convenience, here is the link again.

RawDigger histograms: Part 1. What is the raw data histogram? | RawDigger

Probably the most useful thing I took from the linked article is that the in-camera histogram as well as those displayed by most PP software is wildly dissimilar to an actual plot, particularly for high values.

It should probably also be emphasized that the data in the RAW file is not the same as the sensor capture data (per pixel response). The camera's image processor has already had a go at it...filling in "blanks", smoothing things over, that sort of thing.


Steve

[stevebrot]

At least on a good RAW converter, it is a simple charting of the RAW data

Which converter does that? I may have been operating under a delusion! It was my understanding that ACR and other converters always displayed interpreted (converted) results such that the displayed image and histogram agree.

I will have to do some additional research!


Steve

[stevebrot]

As noted in the comments above, you can construct a histogram directly from the data in the PEF/DNG file. The article posted by BrianR covers the subject in detail and provides a good discussion of how to use the information. For convenience, here is the link again.

For those who are particularly interested in the topic, parts 2 & 3 of the article:

RawDigger Histograms: Part 2. Histogram display modes | RawDigger

RawDigger Histograms, part 3: Overexposure Shapes | RawDigger

The second one would be of interest for the ETTR crowd.


Steve

[stevebrot]

I don't agree with that, I'd say it's differently useful.

Yes, differently useful.

I have been looking around the RawDigger Web site. It is a cool tool, though I am not sure what the practical application might be for the photographer, particularly in the field. For people doing camera reviews and such, people hacking tweakable systems, and software developers, I can see some definitely utility.


Steve

[BrianR]

I have been looking around the RawDigger Web site. It is a cool tool, though I am not sure what the practical application might be for the photographer, particularly in the field. For people doing camera reviews and such, people hacking tweakable systems, and software developers, I can see some definitely utility.

Seems like it would be handy (in the field) to take the cameras jpeg engine out of the histogram equation. It seems this way at least, I'd need to use such a camera in practice to know for sure. Having the option would be nice to try. I'd be willing to take a kick in the pants for response time if a raw histogram takes the camera longer to generate.

At home, where I have access to the raw converter I'll be using, it's less useful (to me, since I'm not one of the people you've mentioned).

Is the histogram displayed by the RAW converter strictly of the data itself or is it based on how the software interprets the data into a displayed image (is that a JPEG or a TIFF?) which might be different than the camera's interpretation?

In my raw converter (Lightroom/ACR), and I suspect most all of them, it's based on the displayed image not the raw data directly (otherwise the histogram wouldn't move when you move the sliders around). So basically you're looking at a different sent of conversion options than the camera has.

In other words, is there really such a thing as a histogram of RAW data?

Yes there is. Head to the RawDigger links.

[stevebrot]

I'd be willing to take a kick in the pants for response time if a raw histogram takes the camera longer to generate.

The problem with using the RAW histogram for in-camera display is that viewing the high end of the range requires significant amount of zoom. This is clear from the RawDigger articles and screen captures. I am not saying it would be impossible to accomplish but the big question might be why bother. ETTR is not some sort of Holy Grail. The same is true of the Zone System or any other technical approach to image capture using media with restricted dynamic range. A good eye and intuition are often adequate.

The existing JPEG-driven in-camera histogram is useful, as is the preview image and exposure warning flashing thingy. A good rule of thumb in the field might be that if one has reason to believe that highlights might be clipping (histogram, visual impression of subject, etc.), chances are that backing off on the exposure might be desirable if retention of near-white detail is needed. Another approach would to simply add -2 EC and bracket.


Steve

[BrianR]

The problem with using the RAW histogram for in-camera display is that viewing the high end of the range requires significant amount of zoom. This is clear from the RawDigger articles and screen captures.

There's no reason to use a linear scale on the x-axis like they do in their article. Their program has a free trial, it's an interesting thing to poke around with.

I am not saying it would be impossible to accomplish but the big question might be why bother. ETTR is not some sort of Holy Grail. The same is true of the Zone System or any other technical approach to image capture using media with restricted dynamic range. A good eye and intuition are often adequate.

The existing JPEG-driven in-camera histogram is useful, as is the preview image and exposure warning flashing thingy. A good rule of thumb in the field might be that if one has reason to believe that highlights might be clipping (histogram, visual impression of subject, etc.), chances are that backing off on the exposure might be desirable if retention of near-white detail is needed. Another approach would to simply add -2 EC and bracket.

As to "why bother", I think a more accurate idea of what's actually happening in the highlights is still a good thing, even if you're not an ETTR fanatic*. I've come up with my own "rules of thumb" for how roughly how large a blinky area will leave me with a workable file, but I think something more predictable would make working with a new camera somewhat easier. At the very least, having a menu choice between a raw histogram and the jpeg one we have now would have stopped countless threads about which one the camera uses


*I never considered myself one. I did tend to punish the highlights for some shadow detail on my older k100d. With the absurd increase in dynamic range of my newer k5iis, I've considerably changed this practice and finessing the exposure to obtain enough information isn't as touchy. In other words, 2 years ago I would have been adamant about wanting a raw histogram, today I just think it would be a spiffy addition.

[stevebrot]

*I never considered myself one.

Me neither. I too appreciate the ETTR theory, but I prefer to build shadow data with a merged image (HDR without tone mapping) if the subject allows. I use a tool called HDRMerge. The process is analogous to compensating development with B&W film.


Steve