Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
09-06-2016, 08:46 AM   #1
Pentaxian




Join Date: May 2016
Photos: Albums
Posts: 756
PEF or DNG? Is there any difference?

Hi. Not sure if it's been covered. Just wondering if there's any difference shooting these formats? Pef seems a bit less convenient since windows cannot view it in their preview app, which I'm used to using when I'm picking out photos to import into lightroom. Are there any advantages of PEF files over DNG? Or I can pretty much switch to DNG and enjoy the extra convenience?

Thanks!

09-06-2016, 08:52 AM   #2
Pentaxian
D1N0's Avatar

Join Date: May 2012
Location: ---
Photos: Gallery
Posts: 4,102
pef vs dng - Search Results - PentaxForums.com
09-06-2016, 08:53 AM   #3
Site Supporter
jatrax's Avatar

Join Date: May 2010
Location: Oregon
Photos: Gallery | Albums
Posts: 10,634
QuoteOriginally posted by awscreo Quote
Not sure if it's been covered.
Quite a few times.

A long time ago PEF files were significantly smaller than DNG so there was some logic to using PEF. These days the size difference is very small so that has gone away. If you use the Pentax software some camera profile settings will transfer with PEF.

Other than that, there is no reason to use PEF that I can see. And if you use Adobe software there are good reasons to use DNG. In short, unless you have a good reason to use PEF, just use DNG.
09-06-2016, 08:55 AM   #4
Site Supporter
boriscleto's Avatar

Join Date: Dec 2009
Location: Liverpool, NY
Photos: Gallery
Posts: 12,675
Someone will come along and talk about sidecar metadata files blah blah blah. And how it is so much better to shoot in PEF and then convert to DNG yada yada yada.

09-06-2016, 08:56 AM   #5
Pentaxian




Join Date: May 2016
Photos: Albums
Posts: 756
Original Poster
QuoteOriginally posted by jatrax Quote
Quite a few times.

A long time ago PEF files were significantly smaller than DNG so there was some logic to using PEF. These days the size difference is very small so that has gone away. If you use the Pentax software some camera profile settings will transfer with PEF.

Other than that, there is no reason to use PEF that I can see. And if you use Adobe software there are good reasons to use DNG. In short, unless you have a good reason to use PEF, just use DNG.
Hmm. I'm currently using LR and PS, so DNG seems like a better suit for me. But, would I have issues processing Pixel Shift images in SilkyPix? I heard that LR doesn't deal with those files very well, and though I could process them in SilkyPix first, and then export as 16bit tiff into lightroom.
09-06-2016, 09:29 AM   #6
Site Supporter
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 26,901
QuoteOriginally posted by boriscleto Quote
Someone will come along and talk about sidecar metadata files blah blah blah. And how it is so much better to shoot in PEF and then convert to DNG yada yada yada.
And some will come along and talk about sidecar files and say that it only makes a difference if you use Lightroom and save metadata separate from the catalog (i.e. a corner case, but important to some folk).

I personally shoot DNG, though PEF is data and functionally equivalent unless one is using older versions of Adobe Camera Raw/Lightroom that lack support for the camera.* There are advantages to DNG, though conventional logic on the matter is a little fuzzy at best. Probably one of the best references I can offer is the best practices Web site offered by ASMP:

Welcome | Digital Photography Best Practices and Workflow | dpBestflow

Now to the fuzzy part:
  • DNG is a standard...sort of. Adobe has submitted its specification to ISO as an extension of TIFF/EP in 2008, but with no firm traction except committee actions since then.
  • DNG is open...no, it is not. Adobe has published the specification, but community process does not drive the specification.
  • DNG is universal...sort of. Implementations vary between camera vendors and tool vendors. The only real guarantee is that an Adobe-generated DNG will work with Adobe products. In practice, DNG from most (all?) cameras will work with Adobe products** and MAY work with non-Adobe tools. Support for Adobe-generated DNG is limited for non-Adobe tools. Try to open an Adobe-generated DNG with Pentax DCU, for example.
  • DNG is future-proof...conditionally true. As long as Adobe or some other entity is around to maintain the standard and the format is being used, DNG will remain viable. As with any other standard, DNG can and very well may be "orphaned" at some future date. This is problematic since DNG is not an image format per se and even with embedded XMP processing meta-data, is only useful as an archive medium when used with Adobe product to translate the file into a view-able image.
  • Added: DNG offers intrinsic validation support...Starting with v1.2, the DNG specification has provided for a hashed digest (checksum) of the core raw data. Tools may take advantage of that value to validate integrity of that data. The only problem is that the tag is optional and at present is only written into Adobe-generated DNG. DNG from cameras and from conversions by non-Adobe software do not contain the hash as of the date of this comment.
Ok...blah, blah, blah...


Steve

* DNG support for new camera models with non-Adobe tools is reportedly mixed. I cannot comment extensively beyond personal experience with dcraw.
** There have been and continue to be issues with some Pentax-generated DNG and non-Pentax RAW converters. At fault are DNG that contain multiple images for HDR, pixel-shift, and other composites intended for merge by the RAW converter. There is some discussion as to whether such DNG are specification compliant at present.

Last edited by stevebrot; 09-08-2016 at 12:14 PM.
09-06-2016, 09:39 AM   #7
Pentaxian




Join Date: May 2016
Photos: Albums
Posts: 756
Original Poster
QuoteOriginally posted by stevebrot Quote
And some will come along and talk about sidecar files and say that it only makes a difference if you use Lightroom and save metadata separate from the catalog (i.e. a corner case, but important to some folk).

I personally shoot DNG, though PEF is data and functionally equivalent unless one is using older versions of Adobe Camera Raw/Lightroom that lack support for the camera. There are advantages to DNG, though conventional logic on the matter is a little fuzzy at best. Probably one of the best references I can offer is the best practices Web site offered by ASMP:

Welcome | Digital Photography Best Practices and Workflow | dpBestflow

Now to the fuzzy part:
  • DNG is a standard...sort of. Adobe has submitted its specification to ISO as an extension of TIFF/EP in 2008, but with no firm traction except committee actions since then.
  • DNG is open...no, it is not. Adobe has published the specification, but community process does not drive the specification.
  • DNG is universal...sort of. Implementations vary between camera vendors and tool vendors. The only real guarantee is that an Adobe-generated DNG will work with Adobe products. In practice, DNG from most (all?) cameras will work with Adobe products and MAY work with non-Adobe tools. Support for Adobe-generated DNG is limited for non-Adobe tools. Try to open an Adobe-generated DNG with Pentax DCU, for example.
  • DNG is future-proof...conditionally true. As long as Adobe or some other entity is around to maintain the standard and the format is being used, DNG will remain viable. As with any other standard, DNG can and very well may be "orphaned" at some future date. This is problematic since DNG is not an image format per se and even with embedded XMP processing meta-data, is only useful as an archive medium when used with Adobe product to translate the file into a view-able image.
Ok...blah, blah, blah...


Steve
Thank you for taking your time to write this) I'll look into it, see if there's any difference for me personally. Right now I'm just concerned on how to properly process Pixel Shift images, I can live with no easy previews in windows explorer and just import and view images directly in LR.
09-06-2016, 09:51 AM   #8
Site Supporter
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 26,901
QuoteOriginally posted by awscreo Quote
But, would I have issues processing Pixel Shift images in SilkyPix? I heard that LR doesn't deal with those files very well, and though I could process them in SilkyPix first, and then export as 16bit tiff into lightroom.
SilkyPix is not the same as Pentax Digital Camera Utility (PDCU)*. The most recent update to PDCU is your best option at present for pixel-shift processing from the K-1. Lightroom will process the file, but with artifact. Conventional wisdom at present is to use PDCU to generate a TIFF that may be further processed in LR.


Steve

* PDCU is a custom product written by SilkyPix for Ricoh/Pentax.

09-06-2016, 09:53 AM   #9
Site Supporter




Join Date: Nov 2014
Location: South Bend, IN, USA
Photos: Gallery | Albums
Posts: 2,879
QuoteOriginally posted by awscreo Quote
Thank you for taking your time to write this) I'll look into it, see if there's any difference for me personally. Right now I'm just concerned on how to properly process Pixel Shift images, I can live with no easy previews in windows explorer and just import and view images directly in LR.
It would help me personally if someone would explain how a pixel-shift image is saved. What advantage does the photographer receive in being able to process the image apart from considering the resulting pixel as one pixel?
09-06-2016, 01:54 PM   #10
Site Supporter




Join Date: Feb 2014
Location: Vancouver, BC
Posts: 98
QuoteOriginally posted by boriscleto Quote
Someone will come along and talk about sidecar metadata files blah blah blah. And how it is so much better to shoot in PEF and then convert to DNG yada yada yada.
DNG files produced by converting from a PEF in ACR are superior to those shot in-camera because (a) they're smaller and (b) they have checksums.
09-06-2016, 04:09 PM   #11
Loyal Site Supporter




Join Date: Apr 2015
Photos: Albums
Posts: 506
QuoteOriginally posted by reh321 Quote
It would help me personally if someone would explain how a pixel-shift image is saved. What advantage does the photographer receive in being able to process the image apart from considering the resulting pixel as one pixel?
I'm not quite sure what you mean by "considering the resulting pixel as one pixel."

Assuming we're talking RAW (since JPEGs made from PS RAW images in the camera aren't processed by the user) you get the normal range of image adjustments available to other raw images. PDCU can do some post processing motion correction. Something like dcraw can pull out the separate four images that a PS raw file contains, which someone competent with say Photoshop can use to mask motion and maybe do some other stuff. Or you can open them per usual in Lr and say use existing tools to maybe blur some motion artifacts or clone them out. In short the benefit is the same as any other image.

---------- Post added 09-06-16 at 04:20 PM ----------

QuoteOriginally posted by bar_foo Quote
DNG files produced by converting from a PEF in ACR are superior to those shot in-camera because (a) they're smaller and (b) they have checksums.
If you do an Adobe conversion on a DNG it produces that checksum as well. Not sure why they aren't produced in camera, but you don't have to shoot PEF to achieve that.

In my K-3ii the DNGs and PEFs were about the same size, within .1MB. After converting the PEF to DNG it was smaller, like as small as 28MB vs 34MB if I used a medium JPEG preview, and say 31MB if I used full sized. Given the cost of storage that size savings isn't an issue for me, but YMMV.
09-06-2016, 04:42 PM   #12
Site Supporter
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 26,901
QuoteOriginally posted by bar_foo Quote
DNG files produced by converting from a PEF in ACR are superior to those shot in-camera because (a) they're smaller and (b) they have checksums.
The in-camera DNG also have checksums. It is required by the DNG spec and if missing, the file will be rejected by processing software (hash calculated at runtime will not agree with hash from file). Edit: Numerous online references make this or similar assertions, but on reviewing the DNG specification and Adobe resources, it is apparent that the checksum (an MD5 hash of the raw data segment) is optional and at present is only done for Adobe-generated DNG and is not offered by any camera vendor or via software by other than Adobe.


If Adobe-generated DNG from PEF are smaller, it is only because the user chose other than the full-size JPEG preview.


Steve

Last edited by stevebrot; 09-08-2016 at 12:18 PM.
09-06-2016, 04:57 PM   #13
Site Supporter




Join Date: Jan 2007
Location: Conroe, TX USA
Posts: 226
QuoteOriginally posted by Oakland Rob Quote
I'm not quite sure what you mean by "considering the resulting pixel as one pixel."

Assuming we're talking RAW (since JPEGs made from PS RAW images in the camera aren't processed by the user) you get the normal range of image adjustments available to other raw images. PDCU can do some post processing motion correction. Something like dcraw can pull out the separate four images that a PS raw file contains, which someone competent with say Photoshop can use to mask motion and maybe do some other stuff. Or you can open them per usual in Lr and say use existing tools to maybe blur some motion artifacts or clone them out. In short the benefit is the same as any other image.

---------- Post added 09-06-16 at 04:20 PM ----------



If you do an Adobe conversion on a DNG it produces that checksum as well. Not sure why they aren't produced in camera, but you don't have to shoot PEF to achieve that.

In my K-3ii the DNGs and PEFs were about the same size, within .1MB. After converting the PEF to DNG it was smaller, like as small as 28MB vs 34MB if I used a medium JPEG preview, and say 31MB if I used full sized. Given the cost of storage that size savings isn't an issue for me, but YMMV.
I believe that the four files out of dcrawps are interpolated to remove the Bayer pattern thereby removing the ability to composite rather than stack. The interpolated color values will contaminate the output. Because I have not taken dcrawps apart I maybe incorrect. Please let me know if so.
RONC
09-06-2016, 05:00 PM   #14
Site Supporter




Join Date: Nov 2014
Location: South Bend, IN, USA
Photos: Gallery | Albums
Posts: 2,879
QuoteOriginally posted by Oakland Rob Quote
I'm not quite sure what you mean by "considering the resulting pixel as one pixel."

Assuming we're talking RAW (since JPEGs made from PS RAW images in the camera aren't processed by the user) you get the normal range of image adjustments available to other raw images. PDCU can do some post processing motion correction. Something like dcraw can pull out the separate four images that a PS raw file contains, which someone competent with say Photoshop can use to mask motion and maybe do some other stuff. Or you can open them per usual in Lr and say use existing tools to maybe blur some motion artifacts or clone them out. In short the benefit is the same as any other image.
OK, let me try again.

Let's pretend that my Q-7 had pixel shift {sometimes the best learning comes from the most "outrageous" model}.

The sensor has 4000 columns by 2664 rows of sensor elements.
If I ask my computer for the properties of a JPEG file, it will report 4000 x 2664.
If I ask my computer for the properties of a DNG file, it will report 4000 x 2664, and that is exactly what it will contain, a 4000x 2664 array of pixels {drilling down to bytes would just confuse discussion}


Are you saying that the same sensor under pixel shift would result in a DNG file containing four 4000 x 2664 arrays of pixels?

added: note that I'm talking about data only in my question; the DNG file contains all kind of other data, but that is irrelevant to my question
09-06-2016, 05:14 PM   #15
Site Supporter
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 26,901
QuoteOriginally posted by rechmbrs Quote
I believe that the four files out of dcrawps are interpolated to remove the Bayer pattern thereby removing the ability to composite rather than stack. The interpolated color values will contaminate the output. Because I have not taken dcrawps apart I maybe incorrect. Please let me know if so.
If you extract the embedded RAW data (can be done in regular dcraw or dcrawps, the results are identical) you get four quite normal-looking TIFFs based on a standard Bayer-interpolation. Running dcrawps in verbose mode gives a few clues as to what Tom is doing, though a more complete analysis might be done from source or by personal communication with the author, PF user @tomtor.

https://www.pentaxforums.com/forums/172-pentax-k-3/296497-implementing-automa...hot-dcraw.html


Steve

Last edited by stevebrot; 09-06-2016 at 07:05 PM.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
adobe software, camera, compression, dng, dslr, files, image, images, lightroom, motion, pef, photography, pixel, post, ps, shot, size
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
PEF or DNG? rlatjsrud Troubleshooting and Beginner Help 10 03-19-2016 06:01 PM
RAW-- PEF vs DNG... difference? CEWren Pentax K-5 9 07-23-2011 10:34 PM
Difference between DNG and PEF? ismaelg Pentax DSLR Discussion 15 06-06-2011 01:16 PM
Pentax/Honeywell...Is there any difference? pickles Pentax SLR Lens Discussion 8 05-29-2010 09:12 AM
Difference between PEF and DNG formats john_mantz Pentax DSLR Discussion 6 09-25-2007 11:24 AM



All times are GMT -7. The time now is 03:43 PM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top