[falconeye]

If they were uniform, shouldn't the cast then be influenced by the quantum efficiency of the different colours in a CFA (e.g., yellow)?

Don't want to dig too deep here as I am no expert on the matter.

Nevertheless, I think the main component in any understanding of the magenta cast in noise seems to be the so-called "sRGB primaries" as measured by, e.g., CIE-D50 color calibration. Typically, they are like RGB=(2,1,1.5) (cf. DxO for a reference).

Normalized to 255, this becomes (255,127,191) which is magenta (Hue is 330). It has nothing to do with quantum efficiency but the color reception of the human.

Now, if the signal isn't white light, then the cast should emerge. For reasons why the signal isn't from white light, I cannot but speculate.

[Class A]

Don't want to dig too deep here as I am no expert on the matter.

Regarding "expert status": Same here.

Although I'm (again) tired and lack the time to properly investigate, I'm still curious. Please feel free to ignore, if you don't find the following helpful.

Nevertheless, I think the main component in any understanding of the magenta cast in noise seems to be the so-called "sRGB primaries" as measured by, e.g., CIE-D50 color calibration. Typically, they are like RGB=(2,1,1.5) (cf. DxO for a reference).

AFAIC, what you quote ("(2,1,1.5)"), is related to a camera-space to RGB space transformation, but is best regarded as "white balance scales" or "(white balance) RGB multipliers".

These are the factors required to scale the RGB camera channels in such a way that they all amount to the same numerical value, assuming the initial values stem from the sensor response to a white object lit by a D50 illuminant.

One (reconstructed) RGB pixel from a camera illuminated by a CIE-D50 source would, e.g., yield (R=1/2, G=1, B=2/3). After multiplication with the white balance scales (2, 1, 1.5), we get (1, 1, 1), i.e., an achromatic colour. All shades of grey should have R=G=B in an RGB space.

If all illuminants had power spectra that could be transformed into each other by a weighting of the camera-space RGB values then one would only need one set of white balance scales per illuminant and the transformation matrix from camera-space to sRGB space could always be the same. As that's not the case, the camera-space to sRGB space transformation matrix still changes with the spectrum of the illuminant. However, for the purposes of understanding the magenta cast, I think it is more helpful to separate the white balance scales from the colour transformation matrix (which can then assume achromatic input colours to have R=G=B).

It has nothing to do with quantum efficiency but the color reception of the human.

The white balance scales reflect the combination of attenuation by the respective colour filters and the power of the filtered band in the illuminant spectrum. So the QE of the colour filters at least plays a role.

Also, assuming the original scene only contained colours that are a linear combination of the camera-space primaries (and are within the sRGB gamut) then if we properly transform from the camera-space into an sRGB space and view the result on an sRGB output device then the stimulus to the human eye should be equivalent to that of the original scene. I guess it is fair to assume that the magenta cast is not caused by the complications of recording arbitrary colours with the camera space primaries, so sRGB primaries (or any other target space primaries) do not appear to be important but rather the white balance scales (which determine the white point).

Finally, I noticed that the Planckian Black Body Radiator locus is shifted towards magenta with respect to the standard CIE illuminants:

(image from "Developing a RAW photo file 'by hand' - Part 1")

However, that should affect all luminance levels equally whereas the magenta cast only seems to manifest itself at very low luminance levels, which led me to my earlier explanation attempt which could very well be wrong.

[falconeye]

AFAIC, what you quote ("(2,1,1.5)"), is related to a camera-space to RGB space transformation, but is best regarded as "white balance scales" or "(white balance) RGB multipliers".

These are the factors required to scale the RGB camera channels in such a way that they all amount to the same numerical value, assuming the initial values stem from the sensor response to a white object lit by a D50 illuminant.

Again, I fear that my initial proposition was too simple to come across easily. Therefore, I give it a second try.

Wrt what you write "to scale ... channels ... such ... that ... same numerical value, assuming ... a white object lit by a D50 illuminant.":

Maybe, let me use a minimal/simplistic amount of math to clarify.

Assume this calibrated sRGB tristimulus of (t_r, t_g, t_b)=(4,2,3) then for white D50 light, raw converters are told to assume:

rawChannel_i = lightLevel_i / t_i

Of course, the row converter (via an embedded camera profile which is the Bayer matrix color filter processing) inverts this to read:

lightLevel_i = t_i * rawChannel_i

(Note: the full math is more complex as it involves a matrix. I ignore this here for the sake of simplicity as it adds nothing to the discussion.)

My assumption simply is that this is wrong. I assume that the correct relations should read:

rawChannel_i = a_i + lightLevel_i / t_i
lightLevel_i = t_i * (rawChannel_i - a_i)

where a_i is a noise source term and I assume it be constant a_i = a. This noise term can be averaged read noise or thermal noise (I consider shot noise to be part of the light itself, no noise term here). A raw converter cannot actually subtract a_i from the signal as it would most likely kill a proper handling of shot noise in any denoising algorithm. The denoiser would have to be aware of a_i (with possible channel dependency) and failing to do so is what I call the bug in black level processing. Moreover, I am almost sure that a_i is prone to pattern noise like banding, hot pixels and smooth regions of glow.

Therefore, we get a background cast a * t_i added to any signal and it happens to be magenta. It becomes visibly as the foreground signal vanishes.

[Class A]

Maybe, let me use a minimal/simplistic amount of math to clarify.

Thanks, I think we both agree that the full transformation entails a matrix multiplication but that for the discussion it is sufficient to assume a simpler transform.

I tried to express that the simple transform you are assuming can be thought of as being actually being performed prior to the matrix multiplication. In other words, I regard your treatment as being compatible with the idea that there is an initial white balance transform and that the latter is a linear transformation. You argument could then be cast as stating that in contrast to what is being done today, the initial white balancing should involve constants (a_i), not just factors (t_i) (with t_i being the "white balance scales").

Your assumption would explain why one cannot adequately combat magenta noise with regular white balance adjustments as the latter just change the t_i. However, while I have no inside knowledge about what the exact effect of the "Shadows | Tint" slider in ACR's camera calibration tab is, my assumption is that it does something very close to changing the a_i in your equations, if not exactly that.

The slider primarily affects dark tones and it would make sense for it to essentially be a way to address a hue bias from the sensor, i.e,. control the a_i constants. This view appears to be supported by a post from Eric Chan, one of the ACR developers.

So perhaps your hypothesis is already taken care of?
Maybe I'm missing some finer point or am just mistaken in my assumptions about ACR.

A raw converter cannot actually subtract a_i from the signal as it would most likely kill a proper handling of shot noise in any denoising algorithm.

I understand that cutting off noise below a blackpoint that is chosen too high is detrimental to proper denoising.

However, could that not be addressed by first performing a linear white balance transform (using appropriate (non-zero)) and then reintroduce a common (now neutral) bias in order to ensure all values are non-negative? Shouldn't that retain the full noise distribution for latter post-processing while avoiding the colour cast?

Therefore, we get a background cast a * t_i added to any signal and it happens to be magenta.

Sure and it seems obvious to me that the combination of the D50 spectrum and the sensor's different sensitivities to different colours explain why the cast is magenta. Assuming all t_i =1, i.e., no white balancing would be required at all, then even in the presence of non-zero a_i, we wouldn't be seeing magenta noise (assuming all a_i have the same value).

Earlier I made the mistake of only looking at the QE of the colour filters, omitting many other factors and hence was misled about the colour cast to expect. My other speculations about the variance of the green channel being lower, etc. were triggered by my assumption that a simple bias problem could be taken care of by camera calibration panel.

[monochrome]

HMoG. I didn't understand a word but it was fun to read. Did Film industry chemists write each other letters before the Internet?

[goldenarrow]

HMoG. I didn't understand a word but it was fun to read. Did Film industry chemists write each other letters before the Internet?

Quote from a movie: "I like to watch."

[falconeye]

Did Film industry chemists write each other letters before the Internet?

They probably did. There are some famous personal letters written by scientists to each other to think things trough

Actually, to let you participate, have a look at
-> Pentax K-3 II : Measurements - DxOMark

And wonder how its "white balance scale" numbers, rewritten as rgb, is nothing but this ◼︎

the initial white balancing should involve constants (a_i), not just factors (t_i) (with t_i being the "white balance scales").

Yes, that's basically what I tried to say. Still linear, but with non-zero offsets.

The slider primarily affects dark tones and it would make sense for it to essentially be a way to address a hue bias from the sensor, i.e,. control the a_i constants. This view appears to be supported by a post from Eric Chan, one of the ACR developers.

Thanks for this link. Very interesting, I did not know it.

Eric seems to assume that the black clipping may only need a shift between green and magenta, with black clipping being controlled by the black level.

And it does help for sure. However, I have to adjust this on an image per image base which is something I'd rather like to avoid.

Here is what I currently need to do when I heavily push shadows:

• Avoid white balance below 3000K
• Dial down vignetting correction to zero (as it boosts magenta in the corners, same "bug" of assuming zero a)
• Adjust camera calibration shadow tint to like -5
• Add shadow split toning with hue around 150
• Add a little hightlight split toning with hue around 330 to compensate
• Set black clipping to remove remaining artefacts

This often gives me excellent results, up to boosts of 6EV in the shadows even at ISO 800. (with a D800, a K-5/K-3 should be the same around ISO 400).

The shadow tint slider does certainly help. But I'd rather like to have a sensor with zero a_i to start with (which was the initial question).

So perhaps your hypothesis is already taken care of?
Maybe I'm missing some finer point or am just mistaken in my assumptions about ACR.

I understand that cutting off noise below a blackpoint that is chosen too high is detrimental to proper denoising.

However, could that not be addressed by first performing a linear white balance transform (using appropriate (non-zero)) and then reintroduce a common (now neutral) bias in order to ensure all values are non-negative?

There maybe is no easy solution. Theoretically, an image would have to allow for negative pixel values. If displayed at a 1:1 scale, negative values would be clipped to zero. At a smaller scale though, pixels would be averaged (binned) and only the remainder would be clipped to zero.

Assume, at the 1:1 scale with a very noisy image with zero light input, half of the pixels would be rather bright (the other half being negative and appearing black) and the image would appear to have a glow / lack of contrast.

At a smaller scale like 10:1 though, 100 pixels would be binned and their fluctuation is reduced to 1/10, i.e., still half the pixels would appear non-black but much darker than before. Downsampling the image made it darker in the shadows and more contrasty.

In other words, traditional black clipping with no negative pixel values, if done at all and if done properly, would have to depend on the scale of magnification (a smaller black clipping value at smaller scale).

I fear all of this is not properly being taken care of. And some of the black clipping may be done by the sensor itself. Ideally, shadows should have no cast whatever be the white balance. But they do have.

[Class A]

In other words, traditional black clipping with no negative pixel values, if done at all and if done properly, would have to depend on the scale of magnification (a smaller black clipping value at smaller scale).

Yes, I regard this as a case of black clipping interfering with denoising with downscaling being a particular form of denoising.

I think moving the black point upwards to a point within the "noise amplitude" should only be possible after a compression of the signal around the noise average, i.e., scaling down the signal both above and below the noise average, so that the new blackpoint does not cut off any non-zero values.

This could allow using a signal bias, thus avoiding negative pixel values, without implying unsatisfactory black levels/contrast.

I fear all of this is not properly being taken care of.

You are probably right.

At least ACR shows sufficiently many peculiarities in other areas as well that I'm led to believe that the engineering at least partially is lacking an adequate modeling basis. I can only speak to LR 3.6, though, as I never cared about anything Adobe did in later versions.

[pinholecam]

HMoG. I didn't understand a word but it was fun to read. Did Film industry chemists write each other letters before the Internet?

I fell asleep like I always did when the calculus prof started the lecture.

Just hand me the exam question tips at the end of the lecture... ok?

[falconeye]

I fell asleep like I always did when the calculus prof started the lecture.

Well, I know.
But to my defense, it was a discourse rather than a lecture, I even used a colored patch and it all started with this question:

What's the technical reason for this magenta shift?

And I thought some people do actually wonder why noise looks magenta more often than not. And thanks to Class A, I think we now do understand that a little better than before.

[Class A]

...it was a discourse rather than a lecture, I even used a colored patch...

Ha ha, you have to put on much more of a song and dance these days than just showing one illustration.

And thanks to Class A, I think we now do understand that a little better than before.

You are much too kind. I just kept on playing the ball back into your court until I understood the issue a bit better.

I'm happy to be the apprentice as gaining an insight into something that previously was shrouded in mystery is one of the best things one can experience, AFAIC.

[falconeye]

So, as of today, Sony officially launched their A6300 (codenamed A7000).

If the rumor in the OP is correct, this would indicate an August 1 launch date for the new Pentax APSC camera.

Which in reality, would mean a pre-Photokina announcement in mid September.

I don't know or speculate if that rumor was accurate. Just indicating its implication.

[D1N0]

The addition of phase detect AF on the sensor is not much use on a dslr unless it as a video centric camera (which Pentax is definitely not). Or you would need a hybrid VF. The K-3 successor would be better of with a specific dslr sensor focussing on iq.

[ffking]

The K-3 successor would be better of with a specific dslr sensor focussing on iq.

if, as suggested, video is not a priority on the FF (yay!), the logic might be towards a new APS-C where it is?

[rawr]

Let's just wait and see if the new/ upgraded 24MP sensor in the A6300 is any good before deciding that Pentax should adopt it. If it only improves the photo specs by 5-10%, Pentax may not want to use it.

What they could do instead is a D5/D500 style feature migration. Stick with a slightly re-tuned version of the current 24MP sensor, then pour a pile of the advanced tech from the Pentax FF (AF, viewfinder, metering, video? etc) into the new APS-C body.