Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
10-29-2018, 02:08 AM   #1
Senior Member




Join Date: Oct 2009
Location: Pune, India
Photos: Albums
Posts: 114
RAW file format

I have a K20D, yes it still takes good photos I have been using it for the last 9 years!

So far I have been using the DNG format for all my images. The file size is ~23 MB. Just for the heck of it, I tried the PEF format today. Was surprised to see the file size to be ~16 MB! That's a huge saving in disk space, even more acute considering that I have thousands of images. Is there more information stored in the DNG? Or is the PEF format with better compression? More importantly, if I convert all the DNG files to PEF will I lose any information? If not, is there such a converter which can do this in a batch mode?

mj

10-29-2018, 02:32 AM   #2
Loyal Site Supporter
Loyal Site Supporter
pschlute's Avatar

Join Date: Mar 2007
Location: Surrey, UK
Photos: Gallery
Posts: 5,632
QuoteOriginally posted by mmjoshi Quote
That's a huge saving in disk space, even more acute considering that I have thousands of images
Are you kidding ? For every 10,000 raw files you will be saving 70,000 MB. Even usb flash drives can hold more than that these days.
10-29-2018, 02:39 AM   #3
Senior Member




Join Date: Oct 2009
Location: Pune, India
Photos: Albums
Posts: 114
Original Poster
QuoteOriginally posted by pschlute Quote
Are you kidding ? For every 10,000 raw files you will be saving 70,000 MB. Even usb flash drives can hold more than that these days.
OK, agreed. But still why waste space if not gaining anything?
10-29-2018, 02:46 AM   #4
Pentaxian
Dartmoor Dave's Avatar

Join Date: Aug 2012
Location: Dartmoor, UK
Photos: Gallery
Posts: 3,091
PEF and DNG both use lossless compression. The difference in file size is down to differences in the compression methods used (I can't explain the maths of it, but there's bound to be someone here on the forum who can).

Some manufacturers use lossy raw compression, although I've never been able to find out for certain whether Pentax ever has. For example, Canon uses lossy compression in its lower-end cameras but lossless in its high-end models. Sony used to use lossy compression in even its top-of-the-range cameras, until customer demand forced them to offer a lossless option.

10-29-2018, 03:05 AM   #5
Loyal Site Supporter
Loyal Site Supporter
BigMackCam's Avatar

Join Date: Mar 2010
Location: North-East of England
Posts: 18,511
This is a subject I keep coming back to every few months... Should I shoot in the manufacturer's native raw format, or use DNG instead?

For me, the biggest advantage of shooting in the native raw format is that it's guaranteed to contain every little bit of proprietary information the manufacturer intended. The manufacturer was responsible for defining the format specification, and we can be certain that the raw files produced by the camera will abide by that specification. If we shoot in DNG format, we're relying on the manufacturer's interpretation of Adobe's DNG specification.

Conversely, the biggest advantage of shooting DNG is that it's readable by most raw development tools, even if the camera is an obscure or new model (there are some exceptions, however - especially if the software makes it's decision to read a file based on the camera manufacturer and model stored in EXIF data). Although DNG is a commercially-owned specification, it has become the closest thing we have to a manufacturer-agnostic raw format standard, and most software caters for it rather well. For this very reason, even if Adobe should go out of business at some future point, I believe the DNG "standard" would live on.

There are other pros and cons to both native raw and DNG, and I struggle to decide which, on balance, is better. For now, I shoot DNG.

Note that if you shoot in the native raw format, it's always possible to convert to DNG using AdobeDNGConverter. The reverse, however, isn't possible. So that's one possible reason to keep shooting PEFs...
10-29-2018, 03:14 AM   #6
Veteran Member
kh1234567890's Avatar

Join Date: Sep 2010
Location: Manchester, UK
Photos: Gallery
Posts: 2,653
QuoteOriginally posted by Dartmoor Dave Quote
PEF and DNG both use lossless compression. The difference in file size is down to differences in the compression methods used.
There are two variants of the DNG lossless compression method. I suspect that the built-in K20D method was the older and not as efficient one.
10-29-2018, 05:40 AM - 1 Like   #7
Veteran Member




Join Date: Jun 2015
Location: South West UK
Photos: Albums
Posts: 1,493
In the end, DNG will be compatible with the largest amount of software over the longest period of time. PEF is and will be supported by less software for less time. Although there is no shortage of PEF support around, DNG will always be better supported.

10-29-2018, 05:53 AM   #8
Veteran Member
Barry Pearson's Avatar

Join Date: Jul 2013
Location: Stockport
Posts: 964
QuoteOriginally posted by Dartmoor Dave Quote
PEF and DNG both use lossless compression.
At one time, Pentax cameras didn't use the DNG compression option.

I can't remember which was the first Pentax camera to use the DNG compression option.
10-29-2018, 06:03 AM   #9
Moderator
Not a Number's Avatar

Join Date: Mar 2012
Location: Venice, CA
Posts: 9,299
QuoteOriginally posted by BigMackCam Quote
For me, the biggest advantage of shooting in the native raw format is that it's guaranteed to contain every little bit of proprietary information the manufacturer intended. The manufacturer was responsible for defining the format specification, and we can be certain that the raw files produced by the camera will abide by that specification. If we shoot in DNG format, we're relying on the manufacturer's interpretation of Adobe's DNG specification.
That's my rationale for using native Raw files.

There aren't any strict standards for Raw files. Raw files are just containers. The image data should be in a TIF compatible format but again there is no strict standard for this.

Difference in size would be due to compression, amount of auxiliary data and number of thumbnails contained. Some Raw files contain a number of JPEG thumbnails at various resolutions as well as versions when you process the file. PEFs use a "sidecar" file (.XMP) to store the processing settings of a Raw editor and that keeps the size down a bit too.

QuoteOriginally posted by Barry Pearson Quote
At one time, Pentax cameras didn't use the DNG compression option.

I can't remember which was the first Pentax camera to use the DNG compression option.
The K-30/50/500 and K-01 do not support PEF only DNG. Which is probably why they shipped with a limited edition of Silkypix instead of the Pentax Digital Camera Utility.

Last edited by Not a Number; 10-29-2018 at 06:11 AM.
10-29-2018, 06:17 AM - 1 Like   #10
Site Supporter
Site Supporter
newmikey's Avatar

Join Date: May 2007
Photos: Gallery
Posts: 2,149
QuoteOriginally posted by mmjoshi Quote
I have a K20D, yes it still takes good photos I have been using it for the last 9 years!

So far I have been using the DNG format for all my images. The file size is ~23 MB. Just for the heck of it, I tried the PEF format today. Was surprised to see the file size to be ~16 MB! That's a huge saving in disk space, even more acute considering that I have thousands of images. Is there more information stored in the DNG? Or is the PEF format with better compression? More importantly, if I convert all the DNG files to PEF will I lose any information? If not, is there such a converter which can do this in a batch mode?

mj
Without wanting to go near the reason I have always preferred PEF over DNG, the simple truth is that converting from PEF to DNG is a one-way street. The reverse is simply not possible so there is no converter available. I wouldn't worry too much about the filesize difference on the K20D (one of my all-time favourite bodies ever) as all of the newer models deliver compressed DNG and storage come cheap nowadays.

As to the often repeated mantra that DNG support in software will be around longer: any native raw format ever released (and some never officially released) will be supported in perpetuity, along with the DNG format by 3rd-party software. Once decoded, the clock does not get turned back, ever. The only possible reason for proprietary raw formats to ever lose software support would be for Adobe to drop it in which case it's extremely easy to drop Adobe in turn with ample replacements available in abundance.
10-29-2018, 06:42 AM - 1 Like   #11
Veteran Member
Barry Pearson's Avatar

Join Date: Jul 2013
Location: Stockport
Posts: 964
QuoteOriginally posted by newmikey Quote
As to the often repeated mantra that DNG support in software will be around longer: any native raw format ever released (and some never officially released) will be supported in perpetuity, along with the DNG format by 3rd-party software. Once decoded, the clock does not get turned back, ever.
A PEF doesn't fully describe the image. That is why software that supports PEFs from existing cameras has to be updated to support PEFs from new cameras. The software developer has to learn about the extra details that the PEF doesn't hold, and build that into the software.

How are future software developers going to discover what those necessary extra details are that are needed for raw conversions? They may not be able to play around with existing cameras to reverse-engineer those details.

A DNG holds those details in extra metadata that enables software to perform a high-quality raw conversion on a new camera without having to build in extra details for that camera. A huge amount and variety of such details is catered for by DNG. That is why there have so far been just 5 versions of DNG in over 14 years. (1.0.0.0 to 1.4.0.0). And even the K-1-series only needs and outputs 1.2.0.0!

(The current Ricoh GR series uses version 1.3.0.0 because it uses lens-correction opcodes which were introduced in that version).
10-29-2018, 07:00 AM   #12
Site Supporter
Site Supporter
newmikey's Avatar

Join Date: May 2007
Photos: Gallery
Posts: 2,149
QuoteOriginally posted by Barry Pearson Quote
How are future software developers going to discover what those necessary extra details are that are needed for raw conversions? They may not be able to play around with existing cameras to reverse-engineer those details.
Why would a future software developer ever need to start from scratch to begin with for proprietary raw formats which have already been decoded in the past? Why would they need to play around with cameras when all of the files are available from archive? Most of the raw converters in use today update support for new camera raw formats within months of their release, open-source software within weeks or even days. Yours is a bogus argument.
10-29-2018, 07:52 AM - 1 Like   #13
Veteran Member
Barry Pearson's Avatar

Join Date: Jul 2013
Location: Stockport
Posts: 964
QuoteOriginally posted by newmikey Quote
Why would a future software developer ever need to start from scratch to begin with for proprietary raw formats which have already been decoded in the past? Why would they need to play around with cameras when all of the files are available from archive? Most of the raw converters in use today update support for new camera raw formats within months of their release, open-source software within weeks or even days. Yours is a bogus argument.
It is important to distinguish between the generic raw file format such as "PEF", and the specific raw file format such as "PEF from K-1".

The former doesn't pose much of a problem. It is the latter that can cause problems, as your example "support for new camera raw formats within months of their release, open-source software within weeks or even days" indicates.

There will be "PEFs from K-1s" around. Will they be sufficient to decide how to perform high quality raw conversions in the new software, without being able to test by experimenting with a camera?

It may be worth noting the vast number of ways that cameras differ from one-another:
DNG and camera innovation

Perhaps it will be possible by using (say) dcraw. After all, Dave Coffin decided to re-design dcraw to conform to some aspects of DNG. So now some features of DNG, including some DNG metadata values for a large range of cameras, are openly available. So in future software will be able to benefit from DNG metadata when processing non-DNG raw files for cameras supported by dcraw. Perhaps in future many people will use PEF in-camera and (perhaps unknowingly) end up with a raw conversion that relies on DNG metadata values.
10-29-2018, 08:17 AM   #14
Site Supporter
Site Supporter
newmikey's Avatar

Join Date: May 2007
Photos: Gallery
Posts: 2,149
QuoteOriginally posted by Barry Pearson Quote
There will be "PEFs from K-1s" around. Will they be sufficient to decide how to perform high quality raw conversions in the new software, without being able to test by experimenting with a camera?
Uhmmm, what? Do you think those files go stale over time?

QuoteQuote:
So now some features of DNG, including some DNG metadata values for a large range of cameras, are openly available. So in future software will be able to benefit from DNG metadata when processing non-DNG raw files for cameras supported by dcraw. Perhaps in future many people will use PEF in-camera and (perhaps unknowingly) end up with a raw conversion that relies on DNG metadata values.
That makes absolute zero sense whatsoever. I'll drop this discussion as it seems to be a repeat of incoherent statements.

Assuming the camera firmware is capable of generating DNG as well as PEF, both are 100% proprietary to Pentax and both contain metadata that existing Adobe software may not correctly interpret until it is updated. Assuming you take either the PEF and convert it on your computer to DNG, you will loose that proprietary metadata. Any DNG manipulated by software on your PC and restored will likely strip that same proprietary data.

Given a Pentax camera which can use either PEF or DNG natively, both formats will contain hinting and (meta)data which is proprietary to a new camera model, neither format will be converted correctly by 3rd-party software (including Adobe's) until it is updated to take the new features into account notwithstanding the fact that Adobe software may be able to extract some image data early on (although even that has proven troublesome in the past). DNG as a format is no more "futureproof" than any other raw format in use today.

Heck, opensource software got to decoding pixelshift images (both DNG as well as PEF) way before Adobe ever did and still has PS movement detection which is miles ahead of ACR.
10-29-2018, 09:15 AM - 2 Likes   #15
Moderator
Not a Number's Avatar

Join Date: Mar 2012
Location: Venice, CA
Posts: 9,299
Take a PEF file. Convert it to DNG using Adobe's converter. Put the DNG on the SD card in your Pentax camera. The camera won't recognize the DNG no matter what version you convert the DNG too.

Tells you something right there about the "universality" of DNG files.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
adobe, archive, camera, cameras, converter, dng, documentary, dslr, file, format, future, images, information, interpretation, iso, lightroom, mb, models, pef, pentax, photographer, photography, size, software, support, tiff
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
Setting RAW file format as default??? careaga Pentax K-30 & K-50 4 12-04-2015 05:14 PM
What RAW file format do you use? Fenwoodian Digital Processing, Software, and Printing 22 11-16-2015 12:35 PM
One-Push File format JPEG to RAW+ timmijo Pentax K-30 & K-50 8 09-08-2015 06:50 PM
K-3 DNG RAW File Format ddjchemist Pentax K-3 & K-3 II 2 12-31-2014 06:01 PM
RAW file format selection with the K-5 PALADIN85020 Pentax K-5 & K-5 II 25 01-08-2011 09:03 PM



All times are GMT -7. The time now is 03:57 AM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top