[mohb]

In another thread about NR and ISO the implication is that there exists a 'pure' Raw file that shouldn't be altered by the camera (even though it creates it) or PP software that isn't under the control of the photographer. Given that the Holy Grail of some seems to be the elimination of noise without altering anything else in the Raw file is this possible without contravening the laws of physics?

[stevebrot]

In theory, the RAW data represents voltages for each photo site on the sensor. "Pure" RAW would be these data as captured. "Not-so-pure" would be stuff like using the image processor for removing statistical outliers (probable noise), detection/mitigation of clipped values, and revising voltages to accommodate changes made, all before packaging the data and writing the file.


Steve

[photoptimist]

The Holy Grail is an accurate estimate of the levels of red, green, and blue light that are in the scene. What if the some pixels are corrupted in ways known to the camera manufacturer but not readily known to the photographer or their software?

Some specific questions are:

1. If the sensor has a defective pixel that always puts out a 100% value (always white), should that bad value be kept for the "pure" raw file?

2. If the sensor has a defective pixel that always puts out a 0% value (always black), should that bad value be kept for the "pure" raw file?

3. If some pixels on the sensor are only 50% as sensitive as the rest of the pixels (because they are on-sensor PDAF pixels), should their values be corrected or uncorrected (the RAW image is criss-crossed by dark lines)?

4. If every pixel is either a bit more or less sensitive than the spec or photographer's exposure setting (that is, at ISO 100, some pixels act as if they are ISO 90 and others act as if they are ISO 110), should the "pure" raw file have ISO 100 pixel values or ISO-whatever pixel values?

What if "pure" is impure?

[joergens.mi]

Have a look here, there you can see what the Prime IV will / can do


Socionext Europe ? Image Processors - Milbeaut®
https://www.eu.socionext.com/assets/downloads/32/Camera-Solution-Catalogue-AD04-00101-1E.pdf
https://www.eu.socionext.com/assets/downloads/32/Milbeaut_Factsheet_6_12.pdf

[stevebrot]

Have a look here, there you can see what the Prime IV will / can do


Socionext Europe ? Image Processors - Milbeaut®
https://www.eu.socionext.com/assets/downloads/32/Camera-Solution-Catalogue-AD04-00101-1E.pdf
https://www.eu.socionext.com/assets/downloads/32/Milbeaut_Factsheet_6_12.pdf

The last link has the best summary, though I do believe that those features relate to in-camera processing to actual image formats and not to low-level processing of sensor data.


Steve

[jbinpg]

Deep Space Object astro stresses the sensor more tham many other types of photography due to the usually longer exposures needed to image the dim target. In the quest to bring out a faint signal from a galaxy or nebula., many techniques have been developed. If you start from a raw file, you have the best chance of success. But a faint signal will be overwhelmed by amp glow and other in-camera produced noise as well as sensor noise itself. I believe we should be worried about pre-processing of raw data only if it ends up removing the signal we are looking for. Otherwise the camera pre-processing is doing us a favor.

[Alex645]

Given that the Holy Grail of some seems to be the elimination of noise without altering anything else in the Raw file is this possible without contravening the laws of physics?

What if the some pixels are corrupted in ways known to the camera manufacturer but not readily known to the photographer or their software?
What if "pure" is impure?

I think for many of us, we have somehow assumed that RAW files were like unscanned film and upon discovering that even RAW files are processed (albeit minimally) that weʻve been duped and that a more pure state would be "the holy grail".

But even film had to be processed and prints were always a second generation with a great need for some post-processing. I can imagine how an astronomer or bio researcher would want unaltered pixel data, but engineers with human manufacturing and economic realities are dealing with millions of pixels that need software tweaks to correct anomalies. (e.g. Is that a newly found pulsar or a glitch in my sensor?)

My perspective is that RAW files are minimally processed and jpegs are out of the camera processed. No post processing on a RAW file and even in a scientific use, is too RAW for consumption. Post-process a jpeg and you are most likely over cooking it.

I agree with the concerns @photoptimist poses; even if we could extract a purely unprocessed RAW file, the only benefit I could imagine would be seeing the actual flaws of the sensor which could be compared to how effective the camera processing is when creating a traditional DNG or PEF/NEF/CR2, etc.

I apologize in advance if Iʻm missing your point.

[stevebrot]

I think for many of us, we have somehow assumed that RAW files were like unscanned film and upon discovering that even RAW files are processed (albeit minimally) that weʻve been duped and that a more pure state would be "the holy grail".

Unscanned film is hardly without processing. Perhaps you meant undeveloped film.

I consider development to negatives to be analogous to what the low level image processor does. The result represents a persistent impression of light on the medium that is characteristic of that medium and for digital where the "medium" is the combination of sensor and processor.


Steve

[Alex645]

Unscanned film is hardly without processing. Perhaps you meant undeveloped film.

Steve

No, I meant unscanned, but youʻre right, even processed (developed) film is.....processed.

Ever since Joseph Nicéphore Niépce (B&W), James Clerk Maxwell (color), and Steven Sasson, the term RAW is relative and not absolute.

"I put my heart and my soul into my work, and have lost my mind in the process." --Vincent Van Gogh

[Digitalis]

I think for many of us, we have somehow assumed that RAW files were like unscanned film and upon discovering that even RAW files are processed (albeit minimally) that weʻve been duped and that a more pure state would be "the holy grail".

Audiophiles and scotch aficionados have a similar mentality, all music is mixed before it is sold commercially and the sound is coloured by the speakers/headphones they listen to it through*. Blended whisky is widely held to be inferior to single malts from a specific distillery**. This quest for greater purity is often expensive and in the end pretty much an exercise in futility. RAW is the best option for editing as the higher bit depth offers more tonal flexibility than 8-bit Jpeg.

* do you really think every recording studio has $80,000 focal speakers with plasma tweeters hooked up with dual mono Tube amps with sterling silver speaker cables?
** though i'm prepared to admit some blended whiskies can be remarkably good.

[Not a Number]

I would say there is no such thing as an "uncooked" RAW. There is signal processing and NR being done at every level/step from the sensor on.

And speaking of audio, dialing the clock back to the early mid 80's, many of the first generation of audio CDs often had a harsh unnatural sound with clipping and silibance. Most of the problem was that sound/recording engineers were used to equalizing (boosting and reducing) parts of the sound spectrum to overcome noise and unequal dynamic response in analog recording methods and media. Some of it was due to the A/D and D/A converters. It took a little learning curve to know what equalization "profiles" to use and tuning the digital converters.

[photoptimist]

The challenge for the scientific photographer is that no sensor puts out unaltered data. The conversion of photon flux to electron counts or voltage is rife with pixel-by-pixel alterations in terms of sensitivity, dark current, saturation, cross-talk and a litany of different noise sources. Scientific imaging requires good calibration to un-alter the "raw" but impure data to form an accurate estimate of the original scene.

The camera maker's clever chips and algorithms to cook up a "better" raw file will certainly correct some impurities (e.g., removing the black and white speckles of dead and hot pixels) but at the cost of replacing one impurity with another lesser impurity (e.g., anomalous pixel-to-pixel correlations where the camera filled in the bad pixel with adjacent data). The scientific user will still need to to a calibration to detect and adjust for these impurities but then calibration was always requirement because a pure image was never an option.


(P.S. there's also the alterations to the image created by the lens such as vignetting, distortion, aberration, etc. That "comet" in the corner of the astro image may be coma!)

[SSGGeezer]

Audiophiles and scotch aficionados have a similar mentality, all music is mixed before it is sold commercially and the sound is coloured by the speakers/headphones they listen to it through*. Blended whisky is widely held to be inferior to single malts from a specific distillery**. This quest for greater purity is often expensive and in the end pretty much an exercise in futility. RAW is the best option for editing as the higher bit depth offers more tonal flexibility than 8-bit Jpeg.

* do you really think every recording studio has $80,000 focal speakers with plasma tweeters hooked up with dual mono Tube amps with sterling silver speaker cables?
** though i'm prepared to admit some blended whiskies can be remarkably good.

And the standard speakers sitting next to the mixing board for years were not very good Radio Shack Tandys so they could estimate how the mix sounded on the average person's gear, in other words, weak equipment. Never been one for Scotch though, but still love an occasional quality Kentucky Bourbon.

[clickclick]

I can't help but think of people talking about their like of colors from CCD sensors and attempting to duplicate the color rendition in post for CMOS sensors. So right there we plainly see that the characteristics of sensor affect the RAW data, and there really can't be a single pure RAW, or they would all look alike. I'm thinking the sensor and camera wrapped around it should be thought of as making a film selection. You know what the baseline characteristic of Plus-X is. You know what happens if you push it. Select your film and work within it's parameters for what you need to do and do not worry too much about "pure" RAW, as it doesn't really exist.

[PDL]

Here is an interesting video that I refered to in another thread - it has even made it to the front page of DPR.

You Tube

You Tube

Your mileage might vary, but it is an interesting discussion on what occurs on a digital processor. Now, imagine how all that "data" would be stored into a file. You can also check out the site supporting ExifTool in which you can bury yourself in file level details.

ExifTool by Phil Harvey

As for Scotch Whiskey, neat and smokey - for gods sake no ice. Bourbon, yup, especially in a Manhattan at Oliver's http://www.mayflowerpark.com/olivers-lounge . Irish Whiskey - I prefer Red Breast, Canadian Whiskey ----- I haven't had a drink of that in twenty years. I prefer Scotch Whiskey --- sipping whiskey. In fact I am going to have dinner with two wee drams about right now.....