[beholder3]

While it is already known for a while that you can not compare a Sony lens shot on a Sony body with any other systems lenses shot on another body due to Sony's major raw cokking of at least noise and vignetting (see KP custom，DFA*85/1.4 CP+ Exhibit - Page 4 - PentaxForums.com), I just stumbled upon a similar situation with Nikon Z as stated by Dxomark when testing the 24-70/2.8:

At the extreme ends of the focal length range, there’s noticeable distortion. However, this is largely corrected by the built-in raw profile that is applied when images are processed. There are also in-camera corrections that can be applied to JPEGs. When these are turned off, the distortion is visible in jpeg images of architecture or similar scenes that have a strong linear element, but simultaneously-captured RAW files are corrected.

Cool, another brand with raw files you can not trust anymore for comparisons.

[biz-engineer]

Of course , all mirrorless camera with PDAF matrix on sensor have to process the raw files to attenuate noise patterns even at moderate ISO setting, simply because half pixels used for AF deliver more noise than full pixels. The areas where AF pixels are located become visible because the noise is more pronounced for those AF areas. If the noise reduction wasn't applied to the raw files on mirrorless camera using on sensor PDAF array, anyone pulling shadows with a third party software would see significant banding. On sensor PDAF is an engineering trade-off, the more sensor area is covered by half pixels, the more effective is AF in low light but also require more noise reduction applied to the raw file.

So, why not correct, on the raw data, lens distortion and aberrations as well, so it makes lenses look great in tests and reviews.

[BigMackCam]

I'm in two minds about this...

On the one hand, as @biz-engineer points out, there's already cooking going on without the lens corrections (there has to be, or the raw files would be terrifying ). And so long as the distortion and vignetting corrections are well implemented, it's one less step required in post-processing - plus, any raw development tool can be used without dependencies on lens profile availability.

On the other hand, it seems like raw should mean raw, with all its warts and scars, or else it should be called something else. Yet, we know that our Pentax camera's pixel mapping function is cooking raw; we know there's some amplification, noise reduction above certain ISOs, and probably even more going on than we realise. But ignorance is bliss, and so long as the raw files provide a good basis for processing and the resulting images look good, does it really matter as much as we think?

If, however, lens performance is assessed and reviewed based on these corrected raw files, I *do* have an issue with that... even though, from an end user perspective, it shouldn't be significant.

[biz-engineer]

Nikon follows Sony in cooking raw files for mirrorless cameras

Processing data from the sensor doesn't mean cooking. It's like for meat, you want a 500 gr of chicken, chicken is prepared, without the guts and feather, so that you can cook it without preparation work. JPEG is like the chicken wings already cooked you can eat, maybe add salt and deep into chili sauce.

---------- Post added 27-08-19 at 22:16 ----------

there has to be, or the raw files would be terrifying

Mirrorless raw files must be processed in order to avoid horses look like zebras and to avoid zebras look like checker boards P.S we don't have this problem with DSLRs.

---

Last edited by biz-engineer; 08-27-2019 at 01:16 PM.

[BigMackCam]

Mirrorless raw files must be processed in order to avoid horses look like zebras and to avoid zebras look like checker boards P.S we don't have this problem with DSLRs.

Ha ha

No, we don't have that specific problem with DSLRs... but we would be naive to think that our raw files are completely un-cooked either. It's the level of cooking that differs

[Rondec]

I'm in two minds about this...

On the one hand, as @biz-engineer points out, there's already cooking going on without the lens corrections (there has to be, or the raw files would be terrifying ). And so long as the distortion and vignetting corrections are well implemented, it's one less step required in post-processing - plus, any raw development tool can be used without dependencies on lens profile availability.

On the other hand, it seems like raw should mean raw, with all its warts and scars, or else it should be called something else. Yet, we know that our Pentax camera's pixel mapping function is cooking raw; we know there's some amplification, noise reduction above certain ISOs, and probably even more going on than we realise. But ignorance is bliss, and so long as the raw files provide a good basis for processing and the resulting images look good, does it really matter as much as we think?

If, however, lens performance is assessed and reviewed based on these corrected raw files, I *do* have an issue with that... even though, from an end user perspective, it shouldn't be significant.

I tend to want less "cooking" when shooting RAW and could care less when shooting jpegs. The problem if, for instance, vignetting is corrected in the RAW, I'm going to run into noise in the corners way faster when I bump the exposure in post than I would otherwise. There is no free lunch.

The only reason that I can see to do this is to have review sites pretend that your lenses are better than they are. "Minimal distortion. The least vignetting we've ever seen with a f1.2 lens!" and so on. Except it is fabricated and since there is no way to shoot these lenses, except on these mirrorless cameras, it isn't even possible to know how much correction is going on behind the scenes.

[BigMackCam]

I tend to want less "cooking" when shooting RAW and could care less when shooting jpegs. The problem if, for instance, vignetting is corrected in the RAW, I'm going to run into noise in the corners way faster when I bump the exposure in post than I would otherwise. There is no free lunch.

Sure. That said, the raw vignetting correction (we can reasonably assume) is only lifting / recovering information that's there. If we find it too noisy, we can add vignetting in post and end up more-or-less where we'd have been without the correction - no? I'd have thought the vast majority of users would actually benefit from the vignetting correction, whilst the minority that didn't want it can counteract it.

The only reason that I can see to do this is to have review sites pretend that your lenses are better than they are. "Minimal distortion. The least vignetting we've ever seen with a f1.2 lens!" and so on. Except it is fabricated and since there is no way to shoot these lenses, except on these mirrorless cameras, it isn't even possible to know how much correction is going on behind the scenes.

Perhaps I'm being too charitable in thinking the benefit is at least partially aimed at the user rather than test results. I'd like to think it is, but I'm prepared to accept it might instead be for the reasons you mention. If that's the primary or even partial motive, I think it's a poor show. Still, if the raw files look good and respond well to processing, I'm willing to be open-minded about it.

I will say, it's a pain in the butt when I'm processing images for a lens with no profile available in the software I'm using. My choices are to manually correct distortion and vignetting (difficult to be precise - impossible, even, for complex distortion), or spend a lot of time creating profiles of my own when I'd rather be out shooting. That said, for the lens profiles I've created, I tend to deal only with distortion. I typically correct vignetting manually, and only then when I feel it's really necessary, as I quite like a mild vignetting effect on many of my photos.

All of this aside, I'd be happier if there were two options for raw capture... (1) uncorrected raw (unlikely, since the results would probably frighten off a large part of the user base), and (2) corrected raw. Plus, of course, flattened and compressed JPEG. Actually, you can add in one more option... I'd love a TIFF capability in all of my cameras. But that's just me

---

Last edited by BigMackCam; 08-27-2019 at 02:56 PM.

[Breakfastographer]

Of course , all mirrorless camera with PDAF matrix on sensor have to process the raw files to attenuate noise patterns even at moderate ISO setting, simply because half pixels

Half pixels? I was under the impression that full pixels are usually sacrificed. I thought only Canon works with half pixels.

[clackers]

Half pixels? I was under the impression that full pixels are usually sacrificed. I thought only Canon works with half pixels.

Yep, that's what I thought too.

I think somebody worked out that it could be 12 percent of the pixels that don't contribute to the RAW file and values have to be invented to put in them. This is where artifacts can appear.

[biz-engineer]

No, we don't have that specific problem with DSLRs... but we would be naive to think that our raw files are completely un-cooked either. It's the level of cooking that differs

We shouldn't use "cooking" for raw files, as long as the raw files can still be cooked (demosaicing, apply WB, pull shadows and things like that applied to 14bits). As I wrote in earlier thread, I exported raw to 16bits TIFFs but I could never pull shadows on 16bit TIFFs as I could do on 14bits RAWs. Someone mentioned that the TIFFs were already baked, and that was right. So, you can't really use "cooking" to describe how sensor data get processed into RAW files.

---------- Post added 28-08-19 at 07:41 ----------

Half pixels? I was under the impression that full pixels are usually sacrificed. I thought only Canon works with half pixels.

If full pixel masking would work, there would be no need to mask any pixel, hence no artifact, because the masking could be virtual (digital). I believe all PDAF on sensor use arrays of interleaved half masked pixels, with the mask orientation for vertical and horizontal lines. If full pixels were used, AF precision couldn't be as good as pixel pitch. The PDAF artifacts don't show at base ISO because the SNR is good enough for the eyes not to pick up the patterns, at high ISO the noise levels are equalized by processing raw data in camera (noise filtering with adjacent pixels), however when users pull shadows in post, no manufacturer can predict how many stops the user will pull shadows, that's why banding is visible when pulling shadows in LR, but isn't visible at high ISO.

[beholder3]

I tend to want less "cooking" when shooting RAW and could care less when shooting jpegs. The problem if, for instance, vignetting is corrected in the RAW, I'm going to run into noise in the corners way faster when I bump the exposure in post than I would otherwise. There is no free lunch.

^This. There is a tendency to rave about fast lenses on forums and in reviews. But all fast lenses show significant (around 1.5 to 3.5 stops) vignetting when used wide open.
So any correction does stress your dynamic range by lifting shadows by up to 3.5 stops already and that is even before you have decided on exposure correction or dedicated shadow correction.
This is quite a major change of data.

If, however, lens performance is assessed and reviewed based on these corrected raw files, I *do* have an issue with that... even though, from an end user perspective, it shouldn't be significant.

It is just one more nail in the coffin of imatest lens testing based on image files. You are not testing lenses any longer, but you test software's ability to compute images. brings us back to the topic of optical bench testing.

---------- Post added 28th Aug 2019 at 08:09 ----------

I'd love a TIFF capability in all of my cameras. But that's just me

Remember what has already been pre-cooked for you in a typical TIFF:

1. white balance applied shiftig all color information around (and over the edges in some cases)
2. demosaicing. again color information gets computed all over the place. noise is introduced and reduced.
3. gamma curve applied. Gone are the evenly spaced linear values.

And if you do apply distortion correction (which I find much more intrusive than noise or vignetting correction) your little camera computer fully replaced image data in the corner with image data from other areas (loads of pixels).

[Trickortreat]

Sad thing is - Sony seems to be stopping the production of sensors without PDAF so even the DSLRs that have Sony sensors will have to cope with those artifacts in future.

[clackers]

Sad thing is - Sony seems to be stopping the production of sensors without PDAF so even the DSLRs that have Sony sensors will have to cope with those artifacts in future.

You do get to choose the masking when you buy the sensors, as I understand it, Trickortreat, just like you get to choose the filters over the wafer, and the gain behavior out of it.

But the GR III does have it, and K-70 some form of it, too (to support video AF), so Ricoh did say yes. Remember, they do already alter the RAW data with noise reduction in the K-70, KP, GR III and K-1 MkII.

[Trickortreat]

You do get to choose the masking when you buy the sensors, as I understand it, Trickortreat, just like you get to choose the filters over the wafer, and the gain behavior out of it.

Isnt masking just a software algorithm? Can PDAF pixels on sensors be used to capture image data or they are just dead pixels when used with OVF? What do you mean by get to choose the masking? Sony seems to be doing PDAF artifact masking better then Nikon on same sensors used.

[Rondec]

Sure. That said, the raw vignetting correction (we can reasonably assume) is only lifting / recovering information that's there. If we find it too noisy, we can add vignetting in post and end up more-or-less where we'd have been without the correction - no? I'd have thought the vast majority of users would actually benefit from the vignetting correction, whilst the minority that didn't want it can counteract it.



Perhaps I'm being too charitable in thinking the benefit is at least partially aimed at the user rather than test results. I'd like to think it is, but I'm prepared to accept it might instead be for the reasons you mention. If that's the primary or even partial motive, I think it's a poor show. Still, if the raw files look good and respond well to processing, I'm willing to be open-minded about it.

I will say, it's a pain in the butt when I'm processing images for a lens with no profile available in the software I'm using. My choices are to manually correct distortion and vignetting (difficult to be precise - impossible, even, for complex distortion), or spend a lot of time creating profiles of my own when I'd rather be out shooting. That said, for the lens profiles I've created, I tend to deal only with distortion. I typically correct vignetting manually, and only then when I feel it's really necessary, as I quite like a mild vignetting effect on many of my photos.

All of this aside, I'd be happier if there were two options for raw capture... (1) uncorrected raw (unlikely, since the results would probably frighten off a large part of the user base), and (2) corrected raw. Plus, of course, flattened and compressed JPEG. Actually, you can add in one more option... I'd love a TIFF capability in all of my cameras. But that's just me

The lenses we are talking about are modern lenses, meaning that there are profiles available for them in Lightroom. I know you use Darktable and I'm not sure how easy it is to get those and use them there. I know for Raw Therapee it is pretty simple and all of my Pentax lenses have profiles in both Lightroom and RT. I don't think Nikon is massaging their RAW files because there won't be lens profiles easily available in commonly used package like Lightroom, but because they want their lenses to look better than they really are.

Everyone does some massaging of their RAW files, trying to get the best results (less noise, better dynamic range and colors), but to me lens adjustments should not be applied, but certainly if they are, you should be able to turn that off.