Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
03-15-2015, 10:29 AM   #1
Senior Member




Join Date: May 2014
Posts: 115
How RAW are the raw files

I red a few threads about astonishing quality of Sigma DP3 and foveon sensors. And I was quite amazed of its 46MP raw files.
Nevertheless it seem not to be a clear sensore advantage, but some kind of HDR. Now that Olympus have announced same kind of thing with 46MP Raw files Im questioning, how much of that is in camera PP.
Although there are quite few effects in different cameras which are applied to the raw file. Like, same sensor by Sony and Nikon has completely different color sheme, not to mention pentax, which is quite unique with its pentax colors. although Olympus is quite active on changing the raw files, you can apply almost all changes and effects to the raw.
Lightroom and Photoshop apparenty are working on applying that format in the worflow, the question is, will it be although possible on just combinning hdr raw files of any camera, or are those parameters defined by the hardware?
Anyone knowing more on that issue? what should we wait next? what is future development in that?

03-15-2015, 10:55 AM   #2
npc
Veteran Member




Join Date: Apr 2013
Posts: 313
RAW files generally contain two things - the sensor readout data (which ideally should be unmodified) + metadata.
Color profile, white balance - all these are just metadata and can be changed later when developing the file without data loss.
However, sometimes compaies "cook" even the sensor data and apply some processing before it is saved - and these changes cannot be reverted, therefore strictly speaking it is not a "raw" file anymore.
Most notorious for this right now are Sony with their lossy compression applied to raw data.
Others are also suspected to apply at least some level of noise reduction to their raw data - there was a good article from DxO about this.
I'm not 100% but I expect that the High-ISO NR and Slow Shutter Speed NR on Pentax cameras actually do a "dark frame substraction" and this is applied to the raw as well as to the jpeg
03-15-2015, 11:06 AM   #3
Pentaxian
Na Horuk's Avatar

Join Date: Feb 2012
Location: Slovenia, probably
Photos: Gallery | Albums
Posts: 11,183
Yeah, some say that certain cameras (Sony, Nikon?) add a little NR or sharpening to their raw files along, along with the compression. Some older cameras didn't even compress, so all raw files had the same, large size. Anyway, the raw files of Pentax cameras seem to be fairly raw, untouched.
And keep in mind, each software you use will read the raw data in its own way and apply some default processing on its own.
Foveon is a completely different sensor type from the usual bayer sensors of other brands. Bayer has three colours, RGB, and then it interpolates data. Foveon on the other hand captures all three colours at every single pixel. So the end effect is that the foveon sensor records much more data/detail. Foveon sensors have their weaknesses, mainly cost and very high noise at higher ISO.

Edit: Oh, and actual algorithms are secret. We don't know what exactly a camera does with the raw data, we can just notice some indicators and interpret that there are some things happening "behind the courtain"

Last edited by Na Horuk; 03-15-2015 at 12:38 PM.
03-15-2015, 12:34 PM   #4
Veteran Member
abmj's Avatar

Join Date: Apr 2010
Location: Central California
Posts: 600
The Olympus process in the new OMD-EM5mk2 is different. The high definition files are actual multiple exposures, using the sensor shake reduction mechanism to take several slightly different images which are stitched together into one high res image. That process, of course, requires a greater degree of in-camera processing to produce that single, 40+ MP image from the same 16.3 MP sensor found in their flagship cameras.

03-15-2015, 01:44 PM   #5
Senior Member




Join Date: May 2014
Posts: 115
Original Poster
Ah, nice. that high resolution mode with shake reduction, could be implemented by pentax too.
At least i would be more than happy have this technique implemented with 12/16 MP sensors, instead of k3 24MP SOny sensor.
03-15-2015, 02:28 PM   #6
Pentaxian




Join Date: Oct 2009
Location: Far North Qld
Posts: 3,301
QuoteOriginally posted by npc Quote
RAW files generally contain two things - the sensor readout data (which ideally should be unmodified) + metadata.
Technically there's no metadata (EXIF) in RAW, that's contained in the embedded JPEG which is also used for preview.
03-15-2015, 03:20 PM   #7
Senior Member




Join Date: Feb 2014
Photos: Albums
Posts: 133
RAW is simply the actual RAW image data + meta data.. meta data may contains information of the camera, shooting condition, and embedded color profile. It does not contains any manipulation of the image or noise reduction as speaking. Most of people "THOUGHT" camera manufacture cheated by embedded the post-tone process to the RAW file before saved, but that's highly inaccurate. If we reverse the logic, we will know that's just a myth.

1) RAW included noise reduction?

This is highly false statement. While one camera at same given ISO might be cleaner than others and clearly we see some noise has smear effect. It doesn't always have to come from the actual noise reduction technique. Mostly you are seeing is the defects from RAW compression software. When compression calculation done wrong, you don't always guaranteed to have 100% product by decompression. While losing some of the data when decompressed, the image defects might appeared. That's exactly why people believed Sony A7 did some tricks on their RAW, but it's highly false statement these defects were due to noise reduction. Simply put! If these defects were results from noise reductions, it means if we turned "ON" and "OFF" noise reduction option on the camera, we should see the defects going away by turning off the NR option on camera. As the result? I am sorry to debunk this.. basically, defects isn't from NR, because we see it no matter what NR setting was.

Secondly, you might argue "NR" is automatically applied to achieve better noise ratio on given ISO to shine the particular product. NOPE! You are wrong again! NR does not apply to RAW whatsoever! No matter your camera is setting ON or OFF on NR, you are getting the same RAW data when you view your RAW with NR off on the RAW software. It means RAW is RAW.. No one is manipulate anything.. it's all in your head.

2) So why do we see defects? if it's not from noise reduction, where is it from? why is some camera better on given ISO than others? they must've cheated?

Once again.. It's all in your head and manufacture is playing exact card to play with your mind. A given camera is better than other camera on given ISO.. It could be the electronic design, or better yet! cheat you on the ISO reading. They could easily provided ISO64 based camera but sell it as ISO100. Makes your mind believe the same given ISO100 is better than competitors. While we are fully trusted on that fool "ISO" system, you obviously won't compare them at same level and automatically assuming ISO100 is ISO100.. One does better on noise with ISO100 is simply "BETTER" camera than others. But reality is you are comparing ISO64 vs ISO100.. which one gets better noise ratio? The idea is fairly simple.. we got fooled because we were taughted from film school.. ISO100 is ISO100 because same film will yield same result in noise ratio. But back in film days.. we also cheated by covering up the film canister barcode so we can PUSH further on developing.. In digital age, manufacture did the same trick!!! only by reverse of PUSH to give you cleaner image so we were all fooled that camera is either DOING BETTER on given ISO.. Or, the camera RAW cheated on us by post-processing something behind our back!

Deeper on this point, if NR was applied to RAW.. It makes no sense for hardware performance. Although NR calculation isn't much for our daily computer, but it sure will be light years heavier to the camera hardware. The reason it's much easier to apply NR on JPEG is simply because of the data size. We are talking about 8bit per channel on JPEG vs 10, 12, 14, 16bit true 3 layers of data on RAW. Just by looking at file size.. Assuming you are looking at a 8 megabyte JPEG image size vs a 36 megabyte RAW image. Assuming calculation requires triple of memory buffer and hardware to process a noise reduction. (I only calculated the memory resource required from photoshop by looking at memory increase from my computer monitor) You are looking at respectfully 24mp vs 108mb per image data on fly. Image processing is differ than video processing because video is a straight pipeline I/O but image processing is more of pure raw CPU floating point calculation.. which means your camera needs 108mb just for 1 image process).. Do you honestly believed that your camera can handle these kinda task to do noise reduction (RAW) on-fly before save? Not to mention 8/36 mb was taking value from a 12 megapixel camera. Think of 36 megapixel camera today.. how much hardware resource to actually doing a 16bit true 3 layer RAW pure data noise reduction? You must've been kidding me then!! What kind of theory is that? Whoever said that obviously have no simple understanding of how computer hardware and calculation works at all!

So the answer of people seem to guess that RAW contains NR information is highly false.. the debunk comes from Sony A7 RAW.. published by Lloyd Chambers in his blog. That's where the noise came up at first. But later debunked by RawDigger: detecting posterization in SONY cRAW/ARW2 files | RawDigger prooved my theory.. It's simply the RAW "COMPRESSION" errores. It's too bad Sony can not save into different RAW type but Pentax DSLR can. In fact, you should probably see better image quality by using DNG simply due to file size is larger so the compression is less aggressive.


Sorry for the long reply.. all I want to say is.. "RAW is RAW".. whoever told you otherwise they got brain washed or they saw some defects from their images then widely spread the rumor. In the actual "limited hardware resource" devices like camera.. Camera manufactures need not to cheat you on the RAW images. It's highly inefficient to them and makes no sense to cheat like that. However, lower the base ISO or "higher the base ISO" is what camera manufacture has been doing for a long time. There is no good or bad to it. Noise ratio from ISO64 vs ISO100 is minor enough you can't even tell even you look at a 1600% blow up anyway. As for RAW.. nothing has changed! It's pure data.
03-15-2015, 04:46 PM   #8
npc
Veteran Member




Join Date: Apr 2013
Posts: 313
QuoteOriginally posted by Steve.Ledger Quote
Technically there's no metadata (EXIF) in RAW, that's contained in the embedded JPEG which is also used for preview.
Technically, the embeded JPEG is just part of the RAW file metatadata.

03-15-2015, 05:17 PM   #9
Pentaxian




Join Date: Oct 2009
Location: Far North Qld
Posts: 3,301
Not at all. JPG preview and/or thumbnail is not metadata.
Metadata describes and gives information about other data, it is not a bitmap.

File contents
Raw files contain the information required to produce a viewable image from the camera's sensor data.
The structure of raw files often follows a common pattern:

A short file header which typically contains an indicator of the byte-ordering of the file, a file identifier and an offset into the main file data
Camera sensor metadata which is required to interpret the sensor image data, including the size of the sensor, the attributes of the CFA and its color profile
Image metadata which is required for inclusion in any CMS environment or database. These include the exposure settings, camera/scanner/lens model, date (and, optionally, place) of shoot/scan, authoring information and other. Some raw files contain a standardized metadata section with data in Exif format.
An image thumbnail
Optionally a reduced-size image in JPEG format, which can be used for a quick preview
In the case of motion picture film scans, either the timecode, keycode or frame number in the file sequence which represents the frame sequence in a scanned reel. This item allows the file to be ordered in a frame sequence (without relying on its filename).
The sensor image data
03-15-2015, 05:22 PM   #10
npc
Veteran Member




Join Date: Apr 2013
Posts: 313
QuoteOriginally posted by photodesignch Quote
So the answer of people seem to guess that RAW contains NR information is highly false.. the debunk comes from Sony A7 RAW.. published by Lloyd Chambers in his blog. That's where the noise came up at first. But later debunked by RawDigger: detecting posterization in SONY cRAW/ARW2 files | RawDigger prooved my theory.. It's simply the RAW "COMPRESSION" errores. It's too bad Sony can not save into different RAW type but Pentax DSLR can. In fact, you should probably see better image quality by using DNG simply due to file size is larger so the compression is less aggressive.
There's a big difference between "lossy compression" and "lossless compression". RAW data is still "raw" even if compressed, but only as long as it is compressed using a lossless algorithm.

Problem discussed in these articles you mention is not noise reduction applied to the RAW data but exactly what you call "raw compression errors" wichi is the effect of the lossy compression of the raw data. All current Sony cameras use lossy compression for their arw files (11bit+7 or whatever), there's no option to turn that off and get real RAW files.

Noise reduction applied to raw data is different thing and can be detected by analyzing the raw files, one of the dxomark articles descrbing this is here:
Noise reduction - DxOMark
and for Pentax :
http://www.dxomark.com/Reviews/DxOMark-review-for-Pentax-cameras

I think I've read somewhere that Leica (and maybe again Sony) applies color cast corrections to their raw fies, which is yet another case where raw files are not really raw.

---------- Post added 03-15-15 at 05:33 PM ----------

QuoteOriginally posted by Steve.Ledger Quote
Not at all. JPG preview and/or thumbnail is not metadata.
Metadata describes and gives information about other data, it is not a bitmap.
You have really weird understanding about what metada is ... really, it can be anything.

As you say: Metadata describes and gives information about other data - No other restrictions.

It might be embedded in the same file, or in a sidecar file as the .xmp .. some cameras even allow you to record a short wav/mp3 file with your image, where you can leave comments/notes about it.

Embedded JPEG or bitmap image is just a data desribing what the RAW data will look like when it is processed, isn't it?

Last edited by npc; 03-15-2015 at 05:52 PM.
03-16-2015, 04:03 AM   #11
Veteran Member




Join Date: Jun 2008
Location: Flyover America
Posts: 4,469
QuoteOriginally posted by Vitalii Quote
How RAW are the raw files
It's a moot question.

From the time you release the shutter to the time you first bring the RAW file up on your screen the photographer is irrelevant.

After that you have more or less five steps.
1. Is it a keeper?
2. If so what do I want the file to do for me visually?
3. Is there enough data (latitude) in the file to accomplish number 2?
4. If so PP accordingly.
5. Done.

Last edited by wildman; 03-16-2015 at 09:39 AM.
03-16-2015, 04:51 AM   #12
Pentaxian




Join Date: Jun 2009
Posts: 11,913
No 'pure' RAW exists, or is even possible. It's a data stream that is fundamentally transformed and manipulated from beginning to end. How much is a mere matter of degree.

Any RAW file has to be produced through a complex multi-stage process of analog to digital conversions, involving probably 5 different chips in the camera (sensor, image processor, various support chips), each with their processing algorithms and firmware, which will shape and form the image in various ways even before you get to the user-controllable settings for NR etc in the camera menu.

Pic related, a typical processing stream for a digital image (for Sony Exmor):

03-16-2015, 04:52 AM   #13
Pentaxian




Join Date: Oct 2009
Location: Far North Qld
Posts: 3,301
QuoteOriginally posted by npc Quote
You have really weird understanding about what metada is ... really, it can be anything.
No need to try and offend me. If you understand things differently, then so be it.

QuoteOriginally posted by npc Quote
Embedded JPEG or bitmap image is just a data desribing what the RAW data will look like when it is processed, isn't it?
Think about what you just wrote....
JPG is a bitmap by the way.
03-16-2015, 06:53 AM   #14
Veteran Member




Join Date: Jan 2010
Location: Michigan
Photos: Gallery
Posts: 2,207
With X, the metadata of the Pentax camera raw tiffs can be decoded and output like this:
identify -verbose *.dng

The embedded thumb can be shown like this:
exiv2 -et *.dng

However, it seems that the Pentax .dng from my K-01 does not contain a thumb.
The above command says no thumb.
What happens when I download from the camera, the dngs do not initially have a preview.

The file manager (Nautilus) crawls through them and makes little png files in ~/.thumbnails

So after a while, the dngs ( and pef) files display with a preview in the icon.
03-16-2015, 04:32 PM   #15
Veteran Member
emalvick's Avatar

Join Date: Feb 2008
Location: Davis, CA
Photos: Gallery
Posts: 1,641
I would think that if we were going to get truly RAW files (if there could be such a thing), they would be fixed size and a bunch of numbers of which we could interpret as we wish. As it is, they are close to that, we just don't know exactly what the camera's processors are doing. But, if we don't like it, then we should probably get a new camera or live with whatever RAW file we have.

As for what constitutes metadata... it's just data. An image can be part of that data, as can audio, etc. Data isn't fixed to any one data type.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
camera, compression, data, effects, files, hdr, information, metadata, noise, olympus, pentax, photography, reduction, sony
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to open RAW files peterjcb Pentax K-3 19 12-19-2013 04:07 PM
Managing unwanted RAW (DNG) files when shooting in RAW+ Weevil Digital Processing, Software, and Printing 3 03-24-2013 07:43 PM
Pentax K20D - are JPG-settings affecting the RAW files? emtor Pentax DSLR Discussion 6 07-08-2011 11:52 AM
How to save stitched RAW files to a RAW file? HermanLee Digital Processing, Software, and Printing 7 07-09-2010 05:51 PM
Does the Photoshop RAW converter work with Pentax RAW Files? mconder Pentax DSLR Discussion 3 05-15-2008 12:14 PM



All times are GMT -7. The time now is 06:01 AM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top