[awscreo]

Hi. Not sure if it's been covered. Just wondering if there's any difference shooting these formats? Pef seems a bit less convenient since windows cannot view it in their preview app, which I'm used to using when I'm picking out photos to import into lightroom. Are there any advantages of PEF files over DNG? Or I can pretty much switch to DNG and enjoy the extra convenience?

Thanks!

[D1N0]

pef vs dng - Search Results - PentaxForums.com

[jatrax]

Not sure if it's been covered.

Quite a few times.

A long time ago PEF files were significantly smaller than DNG so there was some logic to using PEF. These days the size difference is very small so that has gone away. If you use the Pentax software some camera profile settings will transfer with PEF.

Other than that, there is no reason to use PEF that I can see. And if you use Adobe software there are good reasons to use DNG. In short, unless you have a good reason to use PEF, just use DNG.

[boriscleto]

Someone will come along and talk about sidecar metadata files blah blah blah. And how it is so much better to shoot in PEF and then convert to DNG yada yada yada.

[awscreo]

Quite a few times.

A long time ago PEF files were significantly smaller than DNG so there was some logic to using PEF. These days the size difference is very small so that has gone away. If you use the Pentax software some camera profile settings will transfer with PEF.

Other than that, there is no reason to use PEF that I can see. And if you use Adobe software there are good reasons to use DNG. In short, unless you have a good reason to use PEF, just use DNG.

Hmm. I'm currently using LR and PS, so DNG seems like a better suit for me. But, would I have issues processing Pixel Shift images in SilkyPix? I heard that LR doesn't deal with those files very well, and though I could process them in SilkyPix first, and then export as 16bit tiff into lightroom.

[stevebrot]

Someone will come along and talk about sidecar metadata files blah blah blah. And how it is so much better to shoot in PEF and then convert to DNG yada yada yada.

And some will come along and talk about sidecar files and say that it only makes a difference if you use Lightroom and save metadata separate from the catalog (i.e. a corner case, but important to some folk).

I personally shoot DNG, though PEF is data and functionally equivalent unless one is using older versions of Adobe Camera Raw/Lightroom that lack support for the camera.* There are advantages to DNG, though conventional logic on the matter is a little fuzzy at best. Probably one of the best references I can offer is the best practices Web site offered by ASMP:

Welcome | Digital Photography Best Practices and Workflow | dpBestflow

Now to the fuzzy part:

• DNG is a standard...sort of. Adobe has submitted its specification to ISO as an extension of TIFF/EP in 2008, but with no firm traction except committee actions since then.
• DNG is open...no, it is not. Adobe has published the specification, but community process does not drive the specification.
• DNG is universal...sort of. Implementations vary between camera vendors and tool vendors. The only real guarantee is that an Adobe-generated DNG will work with Adobe products. In practice, DNG from most (all?) cameras will work with Adobe products** and MAY work with non-Adobe tools. Support for Adobe-generated DNG is limited for non-Adobe tools. Try to open an Adobe-generated DNG with Pentax DCU, for example.
• DNG is future-proof...conditionally true. As long as Adobe or some other entity is around to maintain the standard and the format is being used, DNG will remain viable. As with any other standard, DNG can and very well may be "orphaned" at some future date. This is problematic since DNG is not an image format per se and even with embedded XMP processing meta-data, is only useful as an archive medium when used with Adobe product to translate the file into a view-able image.
• Added: DNG offers intrinsic validation support...Starting with v1.2, the DNG specification has provided for a hashed digest (checksum) of the core raw data. Tools may take advantage of that value to validate integrity of that data. The only problem is that the tag is optional and at present is only written into Adobe-generated DNG. DNG from cameras and from conversions by non-Adobe software do not contain the hash as of the date of this comment.

Ok...blah, blah, blah...


Steve

* DNG support for new camera models with non-Adobe tools is reportedly mixed. I cannot comment extensively beyond personal experience with dcraw.
** There have been and continue to be issues with some Pentax-generated DNG and non-Pentax RAW converters. At fault are DNG that contain multiple images for HDR, pixel-shift, and other composites intended for merge by the RAW converter. There is some discussion as to whether such DNG are specification compliant at present.

[awscreo]

And some will come along and talk about sidecar files and say that it only makes a difference if you use Lightroom and save metadata separate from the catalog (i.e. a corner case, but important to some folk).

I personally shoot DNG, though PEF is data and functionally equivalent unless one is using older versions of Adobe Camera Raw/Lightroom that lack support for the camera. There are advantages to DNG, though conventional logic on the matter is a little fuzzy at best. Probably one of the best references I can offer is the best practices Web site offered by ASMP:

Welcome | Digital Photography Best Practices and Workflow | dpBestflow

Now to the fuzzy part:

• DNG is a standard...sort of. Adobe has submitted its specification to ISO as an extension of TIFF/EP in 2008, but with no firm traction except committee actions since then.
• DNG is open...no, it is not. Adobe has published the specification, but community process does not drive the specification.
• DNG is universal...sort of. Implementations vary between camera vendors and tool vendors. The only real guarantee is that an Adobe-generated DNG will work with Adobe products. In practice, DNG from most (all?) cameras will work with Adobe products and MAY work with non-Adobe tools. Support for Adobe-generated DNG is limited for non-Adobe tools. Try to open an Adobe-generated DNG with Pentax DCU, for example.
• DNG is future-proof...conditionally true. As long as Adobe or some other entity is around to maintain the standard and the format is being used, DNG will remain viable. As with any other standard, DNG can and very well may be "orphaned" at some future date. This is problematic since DNG is not an image format per se and even with embedded XMP processing meta-data, is only useful as an archive medium when used with Adobe product to translate the file into a view-able image.

Ok...blah, blah, blah...


Steve

Thank you for taking your time to write this) I'll look into it, see if there's any difference for me personally. Right now I'm just concerned on how to properly process Pixel Shift images, I can live with no easy previews in windows explorer and just import and view images directly in LR.

[stevebrot]

But, would I have issues processing Pixel Shift images in SilkyPix? I heard that LR doesn't deal with those files very well, and though I could process them in SilkyPix first, and then export as 16bit tiff into lightroom.

SilkyPix is not the same as Pentax Digital Camera Utility (PDCU)*. The most recent update to PDCU is your best option at present for pixel-shift processing from the K-1. Lightroom will process the file, but with artifact. Conventional wisdom at present is to use PDCU to generate a TIFF that may be further processed in LR.


Steve

* PDCU is a custom product written by SilkyPix for Ricoh/Pentax.

[reh321]

Thank you for taking your time to write this) I'll look into it, see if there's any difference for me personally. Right now I'm just concerned on how to properly process Pixel Shift images, I can live with no easy previews in windows explorer and just import and view images directly in LR.

It would help me personally if someone would explain how a pixel-shift image is saved. What advantage does the photographer receive in being able to process the image apart from considering the resulting pixel as one pixel?

[bar_foo]

Someone will come along and talk about sidecar metadata files blah blah blah. And how it is so much better to shoot in PEF and then convert to DNG yada yada yada.

DNG files produced by converting from a PEF in ACR are superior to those shot in-camera because (a) they're smaller and (b) they have checksums.

[Oakland Rob]

It would help me personally if someone would explain how a pixel-shift image is saved. What advantage does the photographer receive in being able to process the image apart from considering the resulting pixel as one pixel?

I'm not quite sure what you mean by "considering the resulting pixel as one pixel."

Assuming we're talking RAW (since JPEGs made from PS RAW images in the camera aren't processed by the user) you get the normal range of image adjustments available to other raw images. PDCU can do some post processing motion correction. Something like dcraw can pull out the separate four images that a PS raw file contains, which someone competent with say Photoshop can use to mask motion and maybe do some other stuff. Or you can open them per usual in Lr and say use existing tools to maybe blur some motion artifacts or clone them out. In short the benefit is the same as any other image.

---------- Post added 09-06-16 at 04:20 PM ----------

DNG files produced by converting from a PEF in ACR are superior to those shot in-camera because (a) they're smaller and (b) they have checksums.

If you do an Adobe conversion on a DNG it produces that checksum as well. Not sure why they aren't produced in camera, but you don't have to shoot PEF to achieve that.

In my K-3ii the DNGs and PEFs were about the same size, within .1MB. After converting the PEF to DNG it was smaller, like as small as 28MB vs 34MB if I used a medium JPEG preview, and say 31MB if I used full sized. Given the cost of storage that size savings isn't an issue for me, but YMMV.

[stevebrot]

DNG files produced by converting from a PEF in ACR are superior to those shot in-camera because (a) they're smaller and (b) they have checksums.

The in-camera DNG also have checksums. It is required by the DNG spec and if missing, the file will be rejected by processing software (hash calculated at runtime will not agree with hash from file). Edit: Numerous online references make this or similar assertions, but on reviewing the DNG specification and Adobe resources, it is apparent that the checksum (an MD5 hash of the raw data segment) is optional and at present is only done for Adobe-generated DNG and is not offered by any camera vendor or via software by other than Adobe.


If Adobe-generated DNG from PEF are smaller, it is only because the user chose other than the full-size JPEG preview.


Steve

[rechmbrs]

I'm not quite sure what you mean by "considering the resulting pixel as one pixel."

Assuming we're talking RAW (since JPEGs made from PS RAW images in the camera aren't processed by the user) you get the normal range of image adjustments available to other raw images. PDCU can do some post processing motion correction. Something like dcraw can pull out the separate four images that a PS raw file contains, which someone competent with say Photoshop can use to mask motion and maybe do some other stuff. Or you can open them per usual in Lr and say use existing tools to maybe blur some motion artifacts or clone them out. In short the benefit is the same as any other image.

---------- Post added 09-06-16 at 04:20 PM ----------



If you do an Adobe conversion on a DNG it produces that checksum as well. Not sure why they aren't produced in camera, but you don't have to shoot PEF to achieve that.

In my K-3ii the DNGs and PEFs were about the same size, within .1MB. After converting the PEF to DNG it was smaller, like as small as 28MB vs 34MB if I used a medium JPEG preview, and say 31MB if I used full sized. Given the cost of storage that size savings isn't an issue for me, but YMMV.

I believe that the four files out of dcrawps are interpolated to remove the Bayer pattern thereby removing the ability to composite rather than stack. The interpolated color values will contaminate the output. Because I have not taken dcrawps apart I maybe incorrect. Please let me know if so.
RONC

[reh321]

I'm not quite sure what you mean by "considering the resulting pixel as one pixel."

Assuming we're talking RAW (since JPEGs made from PS RAW images in the camera aren't processed by the user) you get the normal range of image adjustments available to other raw images. PDCU can do some post processing motion correction. Something like dcraw can pull out the separate four images that a PS raw file contains, which someone competent with say Photoshop can use to mask motion and maybe do some other stuff. Or you can open them per usual in Lr and say use existing tools to maybe blur some motion artifacts or clone them out. In short the benefit is the same as any other image.

OK, let me try again.

Let's pretend that my Q-7 had pixel shift {sometimes the best learning comes from the most "outrageous" model}.

The sensor has 4000 columns by 2664 rows of sensor elements.
If I ask my computer for the properties of a JPEG file, it will report 4000 x 2664.
If I ask my computer for the properties of a DNG file, it will report 4000 x 2664, and that is exactly what it will contain, a 4000x 2664 array of pixels {drilling down to bytes would just confuse discussion}


Are you saying that the same sensor under pixel shift would result in a DNG file containing four 4000 x 2664 arrays of pixels?

added: note that I'm talking about data only in my question; the DNG file contains all kind of other data, but that is irrelevant to my question

[stevebrot]

I believe that the four files out of dcrawps are interpolated to remove the Bayer pattern thereby removing the ability to composite rather than stack. The interpolated color values will contaminate the output. Because I have not taken dcrawps apart I maybe incorrect. Please let me know if so.

If you extract the embedded RAW data (can be done in regular dcraw or dcrawps, the results are identical) you get four quite normal-looking TIFFs based on a standard Bayer-interpolation. Running dcrawps in verbose mode gives a few clues as to what Tom is doing, though a more complete analysis might be done from source or by personal communication with the author, PF user @tomtor.

https://www.pentaxforums.com/forums/172-pentax-k-3/296497-implementing-automa...hot-dcraw.html


Steve