[BarryE]

Do you think there is a chance you just maybe overthinking this?

Yes. Even market leading product photographers who work for international brands do not use bespoke profiles. They trust their lighting skills, Hasselblads and their Eizo/NEC monitors to obtain accurate colours for clients.

[biz-engineer]

Yes. Even market leading product photographers who work for international brands do not use bespoke profiles. They trust their lighting skills, Hasselblads and their Eizo/NEC monitors to obtain accurate colours for clients.

The canned profiles provided my camera manufacturer can be pretty good, especially the ones provided by Hasselblad. The questions is if those profiles are available in post processing software. For Pentax we have PCDU, which works but outdated and extremly slow. In SPD10 Pro, if I change camera model in the exif of to Fuji X-T3 , or Fuji GFX50, all Fuji styles (standard and film sim) appear in color profile drop down menu, there is nothing equivalent from Pentax.

[biz-engineer]

Do you think there is a chance you just maybe overthinking this?For instance what is the purpose of camera profiles?Why do you feel you need a profile, to correct for existing profile shortcoming, to account for mixed lighting or unusual lighting situations?

My goal was to reproduce Pentax default Jpeg style of PDCU , in Raw Therapee, because the pixel shift processing of RT is a lot better, but default color rendering wasn't matching what I see on camera when I shoot.

After some amount of trial and error efforts, I finally obtained a good match between RT output and PDCU output. My learning points are as follow:

1) Pentax styles (Natural, Bright, Landscape, Vibrant etc...) can only be reliably reproduced in RT if the camera Auto settings for JPEG are disabled (such as Auto shadow correct, high light protection, lens corrections etc..).

2) RT automatic tone curve estimation isn't accurate enough because it is based on the RAW embedded JPEG image that doesn't necessarily cover all gradations from pure black to pure white. Don't use RT automatic tone matching and not use RT automatic camera profile that includes a default tone correction curve and baseline parameters.

3) An accurate calibration color checker (e.g X-Rite passport 24, or X-Rite SG) is not necessary (contrary to suggested), because the color matching processes is a differential process, it is about to create a processing layer to close the gap between sets of two images. Instead, an wide range of tones (from pure black to pure white) and hues (many evenly spaced points from 0 to 360) is better than an calibration absolute 24 patch checker. There is no need to calibrate camera colors out from the camera sensor, Ricoh have already calibrated it before shipping the camera to customers. The error is not produced by the camera, the mismatch error is produced by the difference between RT raw conversion engine and Pentax own raw conversion engine.

4) Bracketed exposures are necessary: due to the fact that it is not possible to achieve pure black with a properly exposed color target (X-rite or Inkjet printed), exposure bracketing is necessary (underexpose to have at least one patch with clipped black to set black level of the Pentax tone curve and overexposed exposure to have at least one patch with clipped whites). Precision mid-gray exposure is not necessary, due style matching is a differential correction, not an absolute correction.

5) Global color adjustment first, then individual HSL correction for each color patch ( H(H), S(H), L(H) ). That is for two reasons:
-> the smaller the HSL correction, the less interpolation error for hue that are located between the hues of the color patches
-> HSL fine tuning range is limited

6) Individual hue point corrections: essential is to have the input hue point based on the hue from the hue data of the input raw file, not the hue points measured on the targeted output. Any error of input hue point propagates as hue correction error.

Once all these conditions are met, RT can output Jpegs that are indistinguishable (at least indistinguishable by my eyes) with original Pentax styled images. Then there is the choice between saving adjustments as .pp3 preset or applying adjustment to a HaldClut file used in the film simulation module of RT.

[TonyW]

Pleased to hear that you have achieved your goal, however it seems to me that you have some learning point based on your trial and error that appear to be misleading.

....

1) Pentax styles (Natural, Bright, Landscape, Vibrant etc...) can only be reliably reproduced in RT if the camera Auto settings for JPEG are disabled (such as Auto shadow correct, high light protection, lens corrections etc..).

Certainly if you wish to produce a mimic of a style then shooting raw is the only way. Are you saying that RT automatically applies its version of the JPG camera styles as found in the raw automatically, and of more importance cannot be overidden by you selection of profile?

3) An accurate calibration color checker (e.g X-Rite passport 24, or X-Rite SG) is not necessary (contrary to suggested), because the color matching processes is a differential process, it is about to create a processing layer to close the gap between sets of two images. Instead, an wide range of tones (from pure black to pure white) and hues (many evenly spaced points from 0 to 360) is better than an calibration absolute 24 patch checker. There is no need to calibrate camera colors out from the camera sensor, Ricoh have already calibrated it before shipping the camera to customers. The error is not produced by the camera, the mismatch error is produced by the difference between RT raw conversion engine and Pentax own raw conversion engine.

A colour checker target is necessary for the production of profiles that attempt to correct for varied lighting conditions as already pointed out in this thread by more than one poster.
An application such as X-rite provides is good for that job but is limited twofold. First a profile can only be as accurate as the amount of varied colours on the target. Second, you have no way of viewing the effect on a real world image at within the app. least AFAIK.
It was already pointed out that the better tool for colour matching would be the free Adobe Profile editor. Not only could you use the colour checker targets but you could have adjusted all other parameters such as starting Color Tables used whatever base profile you have and adjust the Hue, Sat and Lightness of individually selected areas of an image (or even a colour checker!) , adjust overall Colour Matrices. Correct me if I am wrong but the impression I had was that you found the idea of the Adobe solution to require too much time, so my guess is you did not really look at it?

4) Bracketed exposures are necessary: due to the fact that it is not possible to achieve pure black with a properly exposed color target (X-rite or Inkjet printed), exposure bracketing is necessary (underexpose to have at least one patch with clipped black to set black level of the Pentax tone curve and overexposed exposure to have at least one patch with clipped whites). Precision mid-gray exposure is not necessary, due style matching is a differential correction, not an absolute correction.

There is a reason that X-Rite did not need to bother with putting either absolute black or absolute whites; they are by definition already clipped. Therefore the colorimetric values for the X-Rite Color checkers do not offer absolute black or absolute white i.e. L* value = 0 or L* value = 100. Instead the colorimetric value for white L* value = 96.54 and for black L* value = 20.

6) Individual hue point corrections: essential is to have the input hue point based on the hue from the hue data of the input raw file, not the hue points measured on the targeted output. Any error of input hue point propagates as hue correction error.

Easily achieved in the Adobe product and others.

Once all these conditions are met, RT can output Jpegs that are indistinguishable (at least indistinguishable by my eyes) with original Pentax styled images. Then there is the choice between saving adjustments as .pp3 preset or applying adjustment to a HaldClut file used in the film simulation module of RT.

So you have now arrived at a total departure from your original post "Creating .dcp color profile for your camera"?

Which was the very reason I questioned if you were overthinking the problem. Still no harm done and good to hear that at last you have what you need

---

Last edited by TonyW; 01-25-2021 at 05:32 AM.

[biz-engineer]

Certainly if you wish to produce a mimic of a style then shooting raw is the only way. Are you saying that RT automatically applies its version of the JPG camera styles as found in the raw automatically, and of more importance cannot be overidden by you selection of profile?

RT doesn't apply any camera specific JPEG profile. When loading a DNG into RT, the tone curve default is a straight line, unless a tone curve is included in the camera .dcp supplied in color management options. For instance, if I set my Pentax K1 Jpeg style to B&W, and load the raw file in RT, RT displays it as a flat color image. Same if I shot with Natural, Landscape, Bright, Vibrant styles set in camera, all raw will by default come out flat out of RT. After RT correct the raw data for camera model variations and light spectum variations, there is still Jpeg rendering to be applied to the corrected raw data in order to produce a jpeg that looks good.

A colour checker target is necessary for the production of profiles that attempt to correct for varied lighting conditions as already pointed out in this thread by more than one poster.

Yes. My misunderstanding at the beginning, as I saw that it was possible to change camera profile in RT, I thought that the default .dcp was the cause for RT producing JPEG that I disliked, but it wasn't. Changing the .dcp has minor effect on JPEG output, especially for images shot in natural light.

Correct me if I am wrong but the impression I had was that you found the idea of the Adobe solution to require too much time, so my guess is you did not really look at it?

I didn't look at it , if Adobe profile editor is for working on the camera profiles, I thought I would have one .dcp (either the standard Pentax K1 dcp or the one made using X-Rite checker+software) and then make RT presets for Natural, Bright, Landscape, Vibrant, Portrait or using film simulation LUTs added as final stage of RAW image processing.

There is a reason that X-Rite did not need to bother with putting either absolute black or absolute whites; they are by definition already clipped. Therefore the colorimetric values for the X-Rite Color checkers do not offer absolute black or absolute white i.e. L* value = 0 or L* value = 100. Instead the colorimetric value for white L* value = 96.54 and for black L* value = 20.

The reason I wrote that Xrite check wouldn't work for black level setting, is because bright light shining over the darkest surface still returns some light , it's not zero. Pure dark can hardly exist on reflective surface even if painted black there is always some light that comes out (unless the printing process is rather special one, I guess). So my idea to "scan" the Pentax curve from 0 to 100% was to bracket exposure to exceed the boundaries of sensor range. This way, I can build a tone precise tone curve from 0% to 100%, which RT auto-tone match doesn't do well (showing differences in shadow areas of JPEG exports).

Easily achieved in the Adobe product and others.

That's possible, I have to try it. The main problem of Jpeg style matching is the automation. I haven't found a software where I can drop a bunch of raw and camera jpeg files that give me a bunch of presets ready to use.


What would be great is a software where we input some JPEGs from Fuji (or Nikon or Canon, or Leica , or Hasselblad) and a DNG file from a Pentax camera, Software that would produce a HSL preset that render like Fuji OOC JPEGs (or Nikon or Canon, or Leica , or Hasselblad).


From what I've seen other people are doing , for film LUTs for instance, they look on display and tweak sliders manually until the digital file look close to a film photograph but it's guesstimated work. If I download several Fuji Acros film simulation LUT from various sources, they all give JPEG that look different (how the film simulation LUT looks basically depends on how talented was the guy who made it... which vary largely from person to person).

So you have now arrived at a total departure from your original post "Creating .dcp color profile for your camera"?

Yes, I continued to post in that thread while I completely changed direction in the meantime , I was too lazy to open another thread I suppose

However, having the discussion in that thread helped me to do more research and to develop my thinking

[house]

RT doesn't apply any camera specific JPEG profile. When loading a DNG into RT, the tone curve default is a straight line, unless a tone curve is included in the camera .dcp supplied in color management options. For instance, if I set my Pentax K1 Jpeg style to B&W, and load the raw file in RT, RT displays it as a flat color image. Same if I shot with Natural, Landscape, Bright, Vibrant styles set in camera, all raw will by default come out flat out of RT. After RT correct the raw data for camera model variations and light spectum variations, there is still Jpeg rendering to be applied to the corrected raw data in order to produce a jpeg that looks good.



Yes. My misunderstanding at the beginning, as I saw that it was possible to change camera profile in RT, I thought that the default .dcp was the cause for RT producing JPEG that I disliked, but it wasn't. Changing the .dcp has minor effect on JPEG output, especially for images shot in natural light.


I didn't look at it , if Adobe profile editor is for working on the camera profiles, I thought I would have one .dcp (either the standard Pentax K1 dcp or the one made using X-Rite checker+software) and then make RT presets for Natural, Bright, Landscape, Vibrant, Portrait or using film simulation LUTs added as final stage of RAW image processing.


The reason I wrote that Xrite check wouldn't work for black level setting, is because bright light shining over the darkest surface still returns some light , it's not zero. Pure dark can hardly exist on reflective surface even if painted black there is always some light that comes out (unless the printing process is rather special one, I guess). So my idea to "scan" the Pentax curve from 0 to 100% was to bracket exposure to exceed the boundaries of sensor range. This way, I can build a tone precise tone curve from 0% to 100%, which RT auto-tone match doesn't do well (showing differences in shadow areas of JPEG exports).


That's possible, I have to try it. The main problem of Jpeg style matching is the automation. I haven't found a software where I can drop a bunch of raw and camera jpeg files that give me a bunch of presets ready to use.


What would be great is a software where we input some JPEGs from Fuji (or Nikon or Canon, or Leica , or Hasselblad) and a DNG file from a Pentax camera, Software that would produce a HSL preset that render like Fuji OOC JPEGs (or Nikon or Canon, or Leica , or Hasselblad).


From what I've seen other people are doing , for film LUTs for instance, they look on display and tweak sliders manually until the digital file look close to a film photograph but it's guesstimated work. If I download several Fuji Acros film simulation LUT from various sources, they all give JPEG that look different (how the film simulation LUT looks basically depends on how talented was the guy who made it... which vary largely from person to person).


Yes, I continued to post in that thread while I completely changed direction in the meantime , I was too lazy to open another thread I suppose

However, having the discussion in that thread helped me to do more research and to develop my thinking

I'm afraid you've lost me in your dcp adventures. You seem to have reached your goal which is great! The way you speak about dcps and other colour issues are unusual for me and seem to contain a few misunderstandings. Never mind if you got there in the end!

[biz-engineer]

The way you speak about dcps and other colour issues are unusual for me and seem to contain a few misunderstandings.

What's a dcp afterall? Is that not a file that defines how to map a specific camera raw data into a color space, under specific lighting?
Where can I find simplified information (not the full DNG specification) about what meta data are in a DNG?

[biz-engineer]

It was already pointed out that the better tool for colour matching would be the free Adobe Profile editor. Not only could you use the colour checker targets but you could have adjusted all other parameters such as starting Color Tables used whatever base profile you have and adjust the Hue, Sat and Lightness of individually selected areas of an image (or even a colour checker!) , adjust overall Colour Matrices. Correct me if I am wrong but the impression I had was that you found the idea of the Adobe solution to require too much time, so my guess is you did not really look at it?

Thanks for pointing this out. I've now looked at Adobe Profile Editor and read the tutorial from Adobe. This is a great tool indeed. The big advantage of the Adobe Profile Editor is that I can generate one profile per look I want to achieve, and then I can reuse .dcp with any raw editor that takes .dcp as input, so I only have to do the HSL matching works one time. The is a color checker matching feature to auto-match DNG profiles of different cameras, but that's not about matching of JPEG styles. Still, Adobe Profile Editor adjustments are manual (sliders, with visual control), not based on numerical matching of HSL from JPEG outputs.

[TonyW]

Just be aware that you may find visual differences in rendering between raw editors
.dcp file compatibility between Lightroom and RawTherapee? - PentaxForums.com

[steephill]

A useful general background read on this topic - Color is a Slippery Trickster | RawDigger

[biz-engineer]

Just be aware that you may find visual differences in rendering between raw editors

Yes, that is because each brand of image processing software are trying to differentiate on output and they can't necessarily copy what other software are doing, maybe for copyrights.
I don't know what rights are associated with image rendering. For instance, is that allowed to copy Fuji film simulations used in Fuji digital camera for commercial use, I'm not sure about that.
I guess if Ricoh would copy in Pentax cameras the exact same Fuji film LUT used in Fuji camera, they'd be in trouble.

[biz-engineer]

A useful general background read on this topic - Color is a Slippery Trickster | RawDigger

The thing is.... it's not a good idea to use a color calibration checker to match Jpeg images, especially is they are sRGB Jpegs.
Half of the color patches of X-Rite colorcheckers (or datacolor) used for camera profiling are outside of sRGB and outside of Adobe RGB color spaces, because those checkers are especially designed to cover as much as the color space that camera sensors can cover. When using an X-Rite colorchecker to match JPEG, the color rendering shifts are compressed at the borders of the limiting color space (sRGB or aRGB) used for the JPEG exports.
A better way is to print on a sheet of paper , a good number of color tiles all within sRGB, it is a lot cheaper than a camera color calibration checker, and it works better.

[TonyW]

The thing is.... it's not a good idea to use a color calibration checker to match Jpeg images, especially is they are sRGB Jpegs.
Half of the color patches of X-Rite colorcheckers (or datacolor) used for camera profiling are outside of sRGB and outside of Adobe RGB color spaces, because those checkers are especially designed to cover as much as the color space that camera sensors can cover. When using an X-Rite colorchecker to match JPEG, the color rendering shifts are compressed at the borders of the limiting color space (sRGB or aRGB) used for the JPEG exports.
A better way is to print on a sheet of paper , a good number of color tiles all within sRGB, it is a lot cheaper than a camera color calibration checker, and it works better.

Where did you get this information from?

I am sorry to say that it is my belief that the information is incorrect on a number of counts:

• sRGB and Adobe RGB are just synthetic editing spaces not based on any real world output device - sRGB is base on a theoretical CRT display with P3 (I think) phosphors
• The colour checker cards and indeed many others are produced using inks none of which is outside of these synthetic spaces. This is the limit of such colour cards - the accuracy of the profile is limited by the amount of colour displayed. In the case of the normal color checker many of the colour represent natural objects, such as human skin tone, foliage and blue sky. EDIT: All patches except Cyan fall within sRGB color gamut and all fall within Adobe RGB
• Want better accuracy wide gamut then you need the Digital SG Colour checker 140 patch including the 24 from the standard colour checker costing around £350. You will still not have anywhere near the gamut that a DSLR can capture and what can be printed with many modern printers able to exceed Adobe RGB gamut in certain areas
• Colorimetric data is published for these cards
• Printing your colour tiles is not really a good answer if you require accurate profiles unless you can also record the exact colorimetric data for each tile using a spectrophotometer

[biz-engineer]

I am sorry to say that it is my belief that the information is incorrect on a number of counts: sRGB and Adobe RGB are just synthetic editing spaces not based on any real world output device - sRGB is base on a theoretical CRT display with P3 (I think) phosphors The colour checker cards and indeed many others are produced using inks none of which is outside of these synthetic spaces. This is the limit of such colour cards - the accuracy of the profile is limited by the amount of colour displayed. In the case of the normal color checker many of the colour represent natural objects, such as human skin tone, foliage and blue sky Want better accuracy wide gamut then you need the Digital SG Colour checker 140 patch including the 24 from the standard colour checker costing around £350. You will still not have anywhere near the gamut that a DSLR can capture and what can be printed with many modern printers able to exceed Adobe RGB gamut in certain areas Colorimetric data is published for these cards Printing your colour tiles is not really a good answer if you require accurate profiles unless you can also record the exact colorimetric data for each tile using a spectrophotometer

Yes but this is not help my process of matching. My process consist of doing color interpolation between two Jpeg outputs. In order to calculate color corrections I do an interpolation on the color response to changes of HSL correction in the RAW developer (RT in my case). I don't do the double guessing correction with my eye like everyone is doing. I take an exposure of a color target with my camera, then I use the camera to convert to one jpeg output file per color style. Then I load the RAW file in RT, add the tone curve, and output a JPEG with RT with zero HSL correction (neutral). Then I extract the HSL values of each color in the JPEG files from the camera and from RT, I compute the difference of HSL values for each color between OOC JPEG and RT neutral JPEG. Based on the sensitivity of HSL correction input parameters in RT, I interpolate to know how much correction should be applied as a function of how much color difference there is between an OOC Jpeg style and the neutral RT JPEG output. I obtain a set of HSL correction for each color and for each in camera JPEG style. Each set of HSL corrections is applied and a RT profile is saved as preset. Once the computed color correction works for one color, I generalize for all other color tiles, so I obtain a preset by computation and not by moving sliders back and forth looking at my display like other people are doing. Now if the colors in the color target are fully saturated (S=100%), the linear interpolation doesn't work because the color space coercion between the working color space in RT and the output color space (aRGB or sRGB) is non linear (perceptual), and if the color output variation function of HSL correction is non linear, the HSL correction computed on the basis of a linear interpolation is not precise. Therefore, the best color target for my job is the one that contain a set of color patches that never exceed the JPEG output color space so that there is no non linear effect taking place due to perceptual rendering intent, so that the color output variation is fully proportional to the variation of HSL parameters in the RT developer.

[TonyW]

Yes but this is not help my process of matching.

Indeed it is not helping your process of colour matching, so the question I have to ask is why are you presenting as fact your subjective view without any objective data from your peers to back this up? E.G:

Half of the color patches of X-Rite colorcheckers (or datacolor) used for camera profiling are outside of sRGB and outside of Adobe RGB color spaces

There is only one colour that falls outside sRGB that is the Cyan patch, but falls within Adobe RGB

I don't do the double guessing correction with my eye like everyone is doing

Why are you quoting as fact what is purely a personal subjective opinion?

I take an exposure of a color target with my camera, then I use the camera to convert to one jpeg output file per color style. Then I load the RAW file in RT, add the tone curve, and output a JPEG with RT with zero HSL correction (neutral). Then I extract the HSL values of each color in the JPEG files from the camera and from RT, I compute the difference of HSL values for each color between OOC JPEG and RT neutral JPEG. Based on the sensitivity of HSL correction input parameters in RT, I interpolate to know how much correction should be applied as a function of how much color difference there is between an OOC Jpeg style and the neutral RT JPEG output. I obtain a set of HSL correction for each color and for each in camera JPEG style. Each set of HSL corrections is applied and a RT profile is saved as preset. Once the computed color correction works for one color, I generalize for all other color tiles, so I obtain a preset by computation and not by moving sliders back and forth looking at my display like other people are doing.

At worst these are voodoo moves and totally unecessary at best you are just making work for yourself without any scientific or colorimetric information - do you have a spectrophotometer for instance to prove your theories?

Therefore, the best color target is the one that contain a set of color patches that never exceed the output color space with the raw converter produce the JPEG output with a perceptual rendering intent, so that the color output variation is fully proportional to the variation of HSL parameters in the RT developer.

You already have a colour target that contains all the colour patches you need - the X-Rite!!

Perceptual rendering intent is totally misleading as that particular intent is dependent on the makers view of what constitutes a 'pleasing' result. The correct rendering intent is one that produces the best or most pleasing visual result to you.

With the greatest respect your process of matching has no basis as far as I can tell on any known colour science you seem to be intent on re inventing the wheel but without the required toolset to publish any objective findings.

In another attempt to help you understand attached are the figures for the X-Rite colour checker. You will note that the values quoted are in sRGB values and more accurately L*a*b* values. Should you wish to plug these into your own version please use the Lab values as these are the correct values in any colour space you care to use sRGB, Adobe RGB and ProPhoto.

I have also attached a colour checker target showing the Lab values. These values will be correct for any colour space. Note do not try to read the values from the actual colour appearance they will be wrong just use the figures quoted e.g. Patch 01 Dark skin L=38, a =12 b=14, Patch 19 White L=96 a=0 b=0. You may note there are some small differences between the two chart figures, this is accounted for due to the fact that small changes made to the values since the left hand figures (GretagMacbeth) and the right hand X-Rite, also further changes made around 2014. The Delta E difference should not be enough to worry about

I am not trying to be harsh or super critical but genuinely trying to help your understanding