[Fcsnt54]

Ok, I thank everyone for their input on this. A lot of this information is going over my head, but I just wanted to clear up a couple of things;

1) The photo itself. I am not trying to get help for editing this image, I am simply using this image as a demonstration of RAW vs JPG. Initially I was intending to show how superior RAW is to Jpg for 'editing breathing room', however I am currently finding the opposite is true.

2) The sliders and values I changed are purposeful, not for the intent of creating a flattering image but rather to explore which file 'cracks' first (in terms of reaching blown highlights/shadows).

3) -2EV was used for the shot as I felt this might prove how much better RAW can be vs Jpg in terms of clawing back shadows.

Apparently normhead didn't take too well to my 'dear norm' comment, for this I apologise. I was trying to be funny, not patronising, my bad. He's not wrong, the Jpg and RAW image when imported into LR do have different exposures before I have adjusted any values whatsoever, I just wanted to be clear that there was no exposure tweaking on my behalf, not anywhere along the process of extracting the Jpg out of the RAW file (from within the K-1).

I'm now pondering if it is this 'uneven' exposure balance between the two images that leads to the RAW struggling more in the below situation;

RAW


JPG


Same adjustments as before except I bumped the Whites from +28 to +60. As you can see the RAW is blown, the Jpg not (or hardly so). To someone trying to convince them to swap from RAW to Jpg this doesn't look very convincing argument does it?

My sliders may be wild and not atypical of an edit, but my understanding of what RAW files can do was of such that you can have the sliders anywhere you like (vs it's Jpg counterpart) and it will be 'struggling' less, coping better with highlights, shadows, whites etc etc.

So, if I'm hearing everyone correctly then what's going on is one of two things (or a combination of the two);

1) The RAW and Jpg are not even starting off on even footing, LR is interpreting the two files differently (upon import) and as such making global adjustments to both is unfair (depending upon the adjustments and which direction the adjustments are favouring the other). I should try RawTherapee (it's the only other RAW editor I have installed).

2) The RAW file may be 'encountering problems sooner' than the Jpg version as the extra data things it has over the Jpg are the things that are triggering the issues i.e. the additional bits of information that the RAW file has over the Jpg are the things we're seeing causing the problem. The Jpg can't trigger the same issue as it simply lacks the data in the file that could get 'triggered'. In some situations it may be beneficial to have those additional bits of information, just not in this case (if the sliders were actually the look the photographer was aiming for). This time the image is better suited to its lossy Jpg counterpart, where it's missing information is actually of help, not hindrance. Push the sliders differently and the opposite could be true.

Is this right?

Because the camera does do the edit for the jpeg, do you have any adjustments made in camera? Ive been finding that the current jpegs much better then the previous versions. Some of the issues that i find is some smoothing done in camera eating up some of the details. Reasons why I personally prefer the raws when i do have the time. In other times i really dont mind the jpegs, they can be pleasing to the eye, and they do have room for some minor edits.

[BruceBanner]

Because the camera does do the edit for the jpeg, do you have any adjustments made in camera? Ive been finding that the current jpegs much better then the previous versions. Some of the issues that i find is some smoothing done in camera eating up some of the details. Reasons why I personally prefer the raws when i do have the time. In other times i really dont mind the jpegs, they can be pleasing to the eye, and they do have room for some minor edits.

No absolutely not. I tried to make this comparison as fair as possible, anything Jpg related was turned off for the shot as well as turned off for the RAW extract. Dull and lifeless Jpg as much as possible.

[UncleVanya]

I really think the only thing you have established is that if you use these two methods you don't get the results expected. What you can't establish is if the two methods used the same logic despite your attempt to keep them similar. Reality is that the controls may not mean the same thing. Turning off options doesn't establish the same baseline. The reason people use vastly more complex files is to showcase the differences and how well you can manipulate things in Raw. I shoot raw+ and set the jpg to small size sometimes thats what I need - sometimes I feel a need to go farther. Mind you I spent several years with no raw capable tools. I know what the differences are from direct personal experience doing both. Raw offers much more control but isn't needed in a large number of shots.

[stevebrot]

1) The photo itself. I am not trying to get help for editing this image, I am simply using this image as a demonstration of RAW vs JPG. Initially I was intending to show how superior RAW is to Jpg for 'editing breathing room', however I am currently finding the opposite is true.

A good approach would have been to shoot RAW+ with the custom image setting at "Natural" and use the JPEG as is for the comparison.

2) The sliders and values I changed are purposeful, not for the intent of creating a flattering image but rather to explore which file 'cracks' first (in terms of reaching blown highlights/shadows).

Where you will initially see "cracks" are with artifact, not blown anything. Technically, a blown highlight is any pixel having an RGB value at 255,255,255 (white) in the highlights. Likewise an unrecoverable low value is any pixel at 0,0,0 (black) in the deep shadows. Both equate to "no usable information" and could represent either higher or lower values in the original scene. Pulling the values up merely explores the tool's ability to invent tonal information. THERE IS NO SUCH THING AS SUB-PIXEL DATA IN THE SHADOWS. If the sensor gets inadequate light, it will register nothing beyond noise.

Artifact, OTOH, is detail beakdown with attempts at processing. Those are usually most evident at steep tonal gradients in the middle ranges that appear with changes to contrast, brightness, or saturation beyond the ability of the editor to invent what is not there.

3) -2EV was used for the shot as I felt this might prove how much better RAW can be vs Jpg in terms of clawing back shadows.

One can't "claw back" what is not there. The RAW capture data includes 14 data bits per color channel. That is 14 bits to represent "no light detected" (black) through "almost nothing" (1,1,1) to true white (blown highlight). If there is deep shadow in the room, placing exposure to -2 stops less than the matrix metering's best guess will likely result in non-recoverable shadows regardless of whether it is converted to JPEG or saved as RAW. Despite Web advice to the contrary, it is generally safest and most flexible to expose to the shadows and let the highlights fall where they may. If the range of values in the scene exceeds 14 stops, then one must decide whether to sacrifice the highlights or the shadows. Fortunately, the matrix metering system will do a good job of taking a compromise stance. If there is doubt about the range of values in the scene, do a bracketed series with the intent of doing HDR in post.

Note: Digital capture is interesting in that there is greater ability for tonal expression in the top two stops before saturation than at the bottom of the range. The bottom four stops above nil, OTOH, have a paucity of gray values (to make things easy); how do the integers 1-8 sound? Yes, that is eight gray tones above black to represent four stops of exposure. The RAW converter deals with this by dithering the values between pixels.


Steve

---

Last edited by stevebrot; 11-17-2018 at 08:22 PM.

[awscreo]

Interesting comment, what evidence do you have to support this view?

Ok, so I found the article that I read a few years back. It's in Russian, so I'll try to quickly break down what the guy is saying.

Basically, he's comparing the LR/ACR raw conversion to an app called RPP (Raw Photo Processor), which is an open source converter, I'm pretty sure it's dead by now in terms of development.
He's got image samples that show comparisons between the two, the RPP seems to create a more natural red color that isn't blown out.

?????? ?? ????? ?????? ??????????? ????????? RPP ? ??????? ????????????: dmitry_novak

He mentions that the white balance is correct (I'm guessing he metered it with a grey card, I believe he's mainly a still life/product photographer).
I'm not sure how this applies to the latest LR/ACR, I don't know if the basic color profiles were further calibrated or not.

Here's a shot I just played around with in Capture one, I could not make it work at all in Lightroom, at all. It's not perfect right now either, but I think with some further work it'll be ok. I actually might switch to Capture One lol, the only thing holding me now is the Photography package value from Adobe (I use Photoshop a lot), but I might just bite the bullet and get the capture one on top of it. The lightroom sample is the weird looking one (again, I couldn't make it work. If you want, I can upload the raw for you to test).

[GUB]

Ok, I thank everyone for their input on this. A lot of this information is going over my head, but I just wanted to clear up a couple of things;

1) The photo itself. I am not trying to get help for editing this image, I am simply using this image as a demonstration of RAW vs JPG. Initially I was intending to show how superior RAW is to Jpg for 'editing breathing room', however I am currently finding the opposite is true.

2) The sliders and values I changed are purposeful, not for the intent of creating a flattering image but rather to explore which file 'cracks' first (in terms of reaching blown highlights/shadows).

3) -2EV was used for the shot as I felt this might prove how much better RAW can be vs Jpg in terms of clawing back shadows.

Apparently normhead didn't take too well to my 'dear norm' comment, for this I apologise. I was trying to be funny, not patronising, my bad. He's not wrong, the Jpg and RAW image when imported into LR do have different exposures before I have adjusted any values whatsoever, I just wanted to be clear that there was no exposure tweaking on my behalf, not anywhere along the process of extracting the Jpg out of the RAW file (from within the K-1).

I'm now pondering if it is this 'uneven' exposure balance between the two images that leads to the RAW struggling more in the below situation;

RAW


JPG


Same adjustments as before except I bumped the Whites from +28 to +60. As you can see the RAW is blown, the Jpg not (or hardly so). To someone trying to convince them to swap from RAW to Jpg this doesn't look very convincing argument does it?

My sliders may be wild and not atypical of an edit, but my understanding of what RAW files can do was of such that you can have the sliders anywhere you like (vs it's Jpg counterpart) and it will be 'struggling' less, coping better with highlights, shadows, whites etc etc.

So, if I'm hearing everyone correctly then what's going on is one of two things (or a combination of the two);

1) The RAW and Jpg are not even starting off on even footing, LR is interpreting the two files differently (upon import) and as such making global adjustments to both is unfair (depending upon the adjustments and which direction the adjustments are favouring the other). I should try RawTherapee (it's the only other RAW editor I have installed).

2) The RAW file may be 'encountering problems sooner' than the Jpg version as the extra data things it has over the Jpg are the things that are triggering the issues i.e. the additional bits of information that the RAW file has over the Jpg are the things we're seeing causing the problem. The Jpg can't trigger the same issue as it simply lacks the data in the file that could get 'triggered'. In some situations it may be beneficial to have those additional bits of information, just not in this case (if the sliders were actually the look the photographer was aiming for). This time the image is better suited to its lossy Jpg counterpart, where it's missing information is actually of help, not hindrance. Push the sliders differently and the opposite could be true.

Is this right?

Yes but image 1 and 2 in the op appear totally identical. That is the unaltered export from the DNG and the incamera jpg. The two of course should be theoretical identical but only if Pentax intended them to be. The changes have happened after that point. And I suspect from the accumulation of your sliders.

[GUB]

Sorry, should have mentioned that I have zero Adobe editor experience. I use darktable on linux and it is trivial to disable the base curve and change input profiles there. I note the posted rawdigger histograms show no signs of clipping.

So do you see the strange extra blip on the histo in Darktable. You need to pull the exposure back a bit to see it - it is a surprising distance from the main body of the histo. And I am not talking about a clipping indicator but a physical part of the histo. It is so small it may not trigger a clipping warning. And it is green. And big enough to cause the difference in that piece of fern where it probably resides.

[Digitalis]

I tried to make this comparison as fair as possible

Both images look awful on my calibrated display, though I will say the RAW holds more tonal information than the JPEG. These two images have different rendering intents and colour profiles* - these things have effects upon tonal mapping which makes any direct comparison impossible. It is well known that Images in the RAW format have higher bit per pixel values than JPGs - this is allows for greater latitude for editing. If you botch the exposure in s 14 bit RAW you will have a greater chance of recovering some data that would have been obliterated in JPEG with its smaller 8 bit tonal range.

I'll also point out we are dealing with the green channel for the foliage which is where 50% of the cameras pixel density resides.

RAW captures on the other hand, are just a collection of numbers (voltages) and have nothing that may be reasonably interpreted as pixels

Digital sensors are actually analogue devices, these voltages are quantized through the ADC. And the qualities** of the ADC's have a tremendous impact on what is recoverable - I just thought I'd mention that for clarity.


* The in-camera processor will interpret white balance, CFA metameric error and channel cross talk differently than another RAW processor, so no matter how hard you try you will end up comparing Apples to Anvils.
** Bit depth,Noise floor, Dark current handling characteristics...etc

[BruceBanner]

A good approach would have been to shoot RAW+ with the custom image setting at "Natural" and use the JPEG as is for the comparison.



Where you will initially see "cracks" are with artifact, not blown anything. Technically, a blown highlight is any pixel having an RGB value at 255,255,255 (white) in the highlights. Likewise an unrecoverable low value is any pixel at 0,0,0 (black) in the deep shadows. Both equate to "no usable information" and could represent either higher or lower values in the original scene. Pulling the values up merely explores the tool's ability to invent tonal information. THERE IS NO SUCH THING AS SUB-PIXEL DATA IN THE SHADOWS. If the sensor gets inadequate light, it will register nothing beyond noise.

Artifact, OTOH, is detail beakdown with attempts at processing. Those are usually most evident at steep tonal gradients in the middle ranges that appear with changes to contrast, brightness, or saturation beyond the ability of the editor to invent what is not there.



One can't "claw back" what is not there. The RAW capture data includes 14 data bits per color channel. That is 14 bits to represent "no light detected" (black) through "almost nothing" (1,1,1) to true white (blown highlight). If there is deep shadow in the room, placing exposure to -2 stops less than the matrix metering's best guess will likely result in non-recoverable shadows regardless of whether it is converted to JPEG or saved as RAW. Despite Web advice to the contrary, it is generally safest and most flexible to expose to the shadows and let the highlights fall where they may. If the range of values in the scene exceeds 14 stops, then one must decide whether to sacrifice the highlights or the shadows. Fortunately, the matrix metering system will do a good job of taking a compromise stance. If there is doubt about the range of values in the scene, do a bracketed series with the intent of doing HDR in post.

Note: Digital capture is interesting in that there is greater ability for tonal expression in the top two stops before saturation than at the bottom of the range. The bottom four stops above nil, OTOH, have a paucity of gray values (to make things easy); how do the integers 1-8 sound? Yes, that is eight gray tones above black to represent four stops of exposure. The RAW converter deals with this by dithering the values between pixels.


Steve

Ok, first up. Once again to be clear, I specified in the first post that I took the shot the RAW shot with "Custom Image Natural, all values 0", nothing altered. Are you suggesting that there is the possibility that the Jpg file can differ when using the K-1's RAW>Jpg copy/extraction method and choosing absolutely nothing extra applied (i.e. using the same as what the camera took at the time) vs from using RAW+? Because I doubt that, it's surely the exactly same end Jpg result, the only difference is generating it at the time of the shot or after via the Playback menu. You only run the risk id you use the RAW>Jpg feature of the camera and accidentally apply something during that creation, no?

Secondly, when we see the red blinkies turn on in LR from bumping exposure, whites or highlights too high, that's surely an indication of hitting the 255, 255, 255 value, no? My latest crop upload a couple of posts back with the whites now at 60, the RAW file is covered with red blinkies (it's just the export doesn't show it, I'd have to do a screen grab for that), vs the Jpg which hardly has any red blinky showing.

Thirdly, I understand one cannot claw back what isn't there, and was the entire point of doing -2ev and doing stuff like -100 highlights and +100 shadows. I was hoping or expecting that extra 6bits of data the RAW file had over the Jpg to be in those shadows and exposure, that when making those sliders for both images the same values that you could 'see more' from the RAW files shadows vs the Jpg counterpart.

Yes but image 1 and 2 in the op appear totally identical. That is the unaltered export from the DNG and the incamera jpg. The two of course should be theoretical identical but only if Pentax intended them to be. The changes have happened after that point. And I suspect from the accumulation of your sliders.

They do not, look more closely. I thought the same at first but another member pointed out that the RAW and Jpg differ, I think the RAW deeper, darker exposed or contrast and the Jpg weaker contrast. So things I don't think are equal from the start, even when you try as you might to generate a Jpg as close to it's RAW mummy

[slartibartfast01]

I have a couple of observations about the JPEG's being produced - first, JPEG isn't an eight-bit standard; it's a twenty-four bit standard, eight bits per color channel, red, green, and blue. Secondly, it isn't necessarily a compressed image - there's no loss of data if you select the largest file size of JPEG the camera can store, because that's the "100%" standard (uncompressed). The K-1 records the same amount of data for both the TIFF version (whether stored as DNG or PEF), as it does for the uncompressed JPEG: L(36M:7360x4912); RAW: (36M:7360x4912) (Specifications | PENTAX K-1 | RICOH IMAGING). I have no reason to believe that it isn't the SAME data. Now, saving the largest uncompressed file size of either type will obviously still be more time consuming than storing compressed JPEGs, since the file sizes are much smaller in the latter, even given the extra processing time required to convert the sensor's data using a compression algorithm. However, it takes half as much time to store one or the other as it does to store both. I don't know why people keep talking about compressed JPEG data, or comparing that to RAW data, which, to my mind, are not relevant to this discussion at all. Comparing apples to oranges.

If raw files contain 14 bits of data per channel and jpeg files contain 8 bits then I cannot understand your theory that a 100 quality jpeg is not lossy. Unless those extra 6 bits contain no data which seems very unlikely.

[jbinpg]

I agree with Doug that the main problem here is that you are comparing output from two different raw converters: the in-camera Ricoh and the Adobe ACR converters. The output will never be identical. A better comparison would be to use the PDCU converter since it apparently has much the same engine as the in-camera one.

[GUB]

They do not, look more closely. I thought the same at first but another member pointed out that the RAW and Jpg differ, I think the RAW deeper, darker exposed or contrast and the Jpg weaker contrast. So things I don't think are equal from the start, even when you try as you might to generate a Jpg as close to it's RAW mummy

The objective way to compare them is via the histograms as I posted in #25. I chose the black window frame as a crop edge so any cropping variation would be close to black on the histo. The black peaks created have a slightly differing height so this changes the scale in each histo. Once you discount this they are as good as identical.
And re the first set of crops in the OP. I took several pixel for pixel colour comparisons from these. (They are after-all the same image and presumably not resized) .How do you explain the approximately 10% boost in green values that the DNG based one has in comparison to the Jpg one. This will not be a defect from the raw - it will be a colour variation somewhere in the processing. This colour boost easily has the potential to clip one compared to the other.

[pschlute]

And re the first set of crops in the OP. I took several pixel for pixel colour comparisons from these. (They are after-all the same image and presumably not resized) .How do you explain the approximately 10% boost in green values that the DNG based one has in comparison to the Jpg one. This will not be a defect from the raw - it will be a colour variation somewhere in the processing. This colour boost easily has the potential to clip one compared to the other.

They are two different colour spaces. The DNG processed in LR is sRGB and the in-camera processed jpeg is AdobeRGB. not ideal when trying to compare.

If raw files contain 14 bits of data per channel and jpeg files contain 8 bits then I cannot understand your theory that a 100 quality jpeg is not lossy. Unless those extra 6 bits contain no data which seems very unlikely.

There's all sorts of confusion over these issued expressed in articles on the InterNet, partly because of assumptions based on earlier standards based on what is now obsolete technology, such as the idea that the JPEG is an "eight-bit standard" and that all JPEG files are necessarily compressed. I've been struggling to figure this stuff out, myself, and found that I'm getting into areas in which precious little information is available. I think the mfgr. assumes that people who just want to take pictures don't need to trouble their pretty li'l heads about the technical mumbo-jumbo.

But I think those numbers refer to two different things - the 12/14 bit analog to digital converters (one per pixel, contained within the sensor) in the cameras are preprocessing the data before any file structure is in the works. The fact that the file formats provide a container for three color channels has nothing to do with the data coming off the sensor, which provides data describing one color channel per pixel. Because the Bayer filter method the sensor uses only provides data on one color per pixel (ignoring "pixel-shift" rendering which effectively multiplies the number of recorded pixels) in an unsigned one byte integer, storing one of 16,384 possible hues for that one color. And the "jpeg files contain 8 bits" is incomplete - JPEG is twenty-four bits, eight bits for each of the three primary colors. They also store three or seven bits per pixel of luminance information (one nybble or byte, respectively, containing a signed binary integer), as I understand it, for a total of twenty-four or thirty-two bits per pixel. I label this as mere opinion at this point, having been unable to locate any solid information.

What I've been unable to locate, and which I feel is the information necessary to really and definitively answer the question, are the technical specifications for the sensor's actual output, the conversion from that output (electrical signals) to both raw and JPEG formats as stored in the camera. That information must be available somewhere, or else software developers wouldn't be able to sell raw data editors.

---------- Post added 2018-11-18 at 09:14 AM ----------

They are two different colour spaces. The DNG processed in LR is sRGB and the in-camera processed jpeg is AdobeRGB. not ideal when trying to compare.

Interesting point. And related to another mystery. Why do the light-capturing devices and software that supports them use the primary colors of pigment, and why do color printers use the primary colors of light? Seems like it should be the other way around - cameras should use CYMK, and printers should use RGB. What am I missing?

---------- Post added 2018-11-18 at 09:27 AM ----------

This whole debate is like one that has been similarly raging among handgun enthusiasts for a long time regarding the optimal caliber, bullet design, and powder load for cartridges carried for personal defense. Which is better, raw or JPEG? I think the answer was best stated by Forrest Gump at the end of the movie: "I think it's both." Or like the Certs commercials from the early '60's - "Stop! You're both right!". Depends on what you want to use the image data for, don't it?

[Digitalis]

cameras should use CYMK, and printers should use RGB

Camera manufacturers have enough trouble managing colour accuracy with the traditional Bayer CFA switching to CMYK would be a disaster.. Professional printers don't use RGB, these days printers go beyond CMYK. The top of the line Epson printers which use theUltraChrome HDX inkset go like this : Matte Black, Photo Black, Light Black, Cyan, Light Cyan, Vivid Magenta, Vivid Light Magenta, Yellow, Orange, Green, Light Light Black,Violet.

Just a trifle more complicated than CMYK, No?