[stevebrot]

It is important to distinguish between the generic raw file format such as "PEF", and the specific raw file format such as "PEF from K-1".

The difference between files for different models is very slight, usually (always?) just the camera name and other stuff in the metadata. Tools such as Dave Coffin's dcraw are usually quite happy working with new model PEF before the camera is added to the compatible list. That does not mean there is not metadata to address special processing needs, only that PEF as a format has not changed in pretty much forever.

Take a PEF file. Convert it to DNG using Adobe's converter. Put the DNG on the SD card in your Pentax camera. The camera won't recognize the DNG no matter what version you convert the DNG too.

Tells you something right there about the "universality" of DNG files.

Yep...DNG is Adobe's proprietary solution to provide a common currency between its various tool offerings. While the specification (not really a standard) is published, an Adobe-generated DNG is not generally not fully readable by non-Adobe products/devices, if at all. The files are DNG-compliant, just not data-compatible.


Steve

[kypfer]

At one time, Pentax cameras didn't use the DNG compression option.

I can't remember which was the first Pentax camera to use the DNG compression option.

If anyone's interested, neither my *istD S nor my *istD L2 appear to use DNG (at least I can't find reference to it in the handbooks or menus).
My K-5 does ... so the functionality would appear to have been introduced at some point between 2006 and 2010, from this limited sample

[StiffLegged]

If anyone's interested, neither my *istD S nor my *istD L2 appear to use DNG (at least I can't find reference to it in the handbooks or menus).
My K-5 does ... so the functionality would appear to have been introduced at some point between 2006 and 2010, from this limited sample

DNG was certainly available with the K-7 and I've used it ever since switching to RAW shooting. Whether it was the first model to use it I don't know, but my old version of CameraRAW reads it without issues or annual payment of Adobe Tax.

[Not a Number]

The K10D supports DNG format.

[BigMackCam]

The K10D supports DNG format.

Indeed. It was among the first cameras from any brand to offer DNG format for raw shots

[Dartmoor Dave]

It was among the first cameras from any brand to offer DNG format for raw shots

And in my humble opinion it remains the camera that has produced the best results from those DNG files at ISO 100.

[BigMackCam]

And in my humble opinion it remains the camera that has produced the best results from those DNG files at ISO 100.

I am biased, but I'd have to agree

[stevebrot]

If anyone's interested, neither my *istD S nor my *istD L2 appear to use DNG (at least I can't find reference to it in the handbooks or menus).

The K10D supports DNG format.

Yep, it debuted on the K10D with Pentax being one of the early makers to offer DNG support.


Steve

[Barry Pearson]

Assuming the camera firmware is capable of generating DNG as well as PEF, both are 100% proprietary to Pentax and both contain metadata that existing Adobe software may not correctly interpret until it is updated. Assuming you take either the PEF and convert it on your computer to DNG, you will loose that proprietary metadata. Any DNG manipulated by software on your PC and restored will likely strip that same proprietary data.

Given a Pentax camera which can use either PEF or DNG natively, both formats will contain hinting and (meta)data which is proprietary to a new camera model, neither format will be converted correctly by 3rd-party software (including Adobe's) until it is updated to take the new features into account notwithstanding the fact that Adobe software may be able to extract some image data early on (although even that has proven troublesome in the past). DNG as a format is no more "futureproof" than any other raw format in use today.

My aim isn't to change what you do. I'm trying to establish some facts that will stand up to scrutiny.

I believe the following is true:
From when Pentax began to use DNG in cameras, they have continued to support DNG in all subsequent cameras that support raw
But the converse does not apply. There have been a number of Pentax cameras that support DNG but not PEF.
Not the major K-Mount cameras, but some nevertheless: Q-Series; K-500 (?); K-30 (?); MX-1 (?). Perhaps others.

Edit: Whoops! I've just that "Not a Number" was there before me!
https://www.pentaxforums.com/forums/6-pentax-dslr-discussion/378298-raw-file...ml#post4501388

My interpretation is that DNG is now the Pentax main and consistent raw file format, and PEF is an optional extra for historical purposes.
(Ricoh uses DNG, not its own raw file format if they ever had one, for its own cameras that support raw. Such as the GR II, and I'm confident for the GR III next year).

When Adobe software handles a DNG from out of a camera, it uses the published DNG metadata in the file, but not any "secret sauce" that is not in the DNG specification.
(What actually happened is that Adobe identified what it needed to do its raw conversions, and designed the DNG metadata to match!)
I believe the DNG format has never been influenced by the needs of a Pentax camera. Those cameras have used DNG "as-is".
(I believe this leaves something to be desired when Pixel-Shift is used. But that appears to apply whether Adobe is processing a PEF or a DNG).

I accept that there is probably some non-Adobe software that supports PEFs (once they have reversed engineered them) but not DNG.
At one time this was typically because they had already done much of the work to support PEFs, and avoided the extra generic work to support DNGs for any cameras.
I'm not sure what reasons they give nowadays.

Something interesting about DNG is the amount of material available free with no questions asked and no Non-Disclosure Agreements.
Obviously the DNG specifications!
But also the DNG SDK. It comes with both executable programs and lots of C++ files. I often use it to have a peek inside out-of-camera DNGs, hence the DNGs from my own Pentax cameras and also the Ricoh GR II.

I've been using DNG for over 14 years. I've published a huge amount about it:
DNG - Digital Negative format
DNG » Barry's blog
In the unlikely event you feel the need to read it, I would welcome being informed of any errors you find.

---------- Post added 30th Oct 2018 at 04:05 PM ----------

.DNG is Adobe's proprietary solution to provide a common currency between its various tool offerings.

Adobe offered DNG to ISO years ago. ISO TC42 WG18 were expected to publish it as an ISO standard. It would then not belong to Adobe in any sense.
For some reason, (my contacts won't say why), this process appears to have stalled.

Adobe did the same with TIFF, (which they "own"). Unfortunately ISO mangled it to become TIFF/EP, which was not prescriptive enough to be useful for interchange. It ended up more as a kit of optional parts.
(DNG and NEF are both based on TIFF/EP. DNG was Adobe's proposal to ISO to make TIFF/EP a more useful standard).

(Adobe also supplied PDF to ISO. It is now, I think, 3 ISO standards, and Adobe don't own it. I think ISO have also taken over XMP, as they presumably needed to for PDF purposes).

---------- Post added 30th Oct 2018 at 04:16 PM ----------

Take a PEF file. Convert it to DNG using Adobe's converter. Put the DNG on the SD card in your Pentax camera. The camera won't recognize the DNG no matter what version you convert the DNG too.

Tells you something right there about the "universality" of DNG files.

I've haven't time to check, but here is my guess:

The DNG Converter puts XMP into the DNG to record what it has done.
Probably the camera simply can't handle the XMP. After all, it is not needed by the camera and doesn't put it in the DNG.
If that is the reason, I don't know why the camera can't ignore it.

There are other possibilities.
Perhaps Pentax cameras write using a subset of DNG, for example with the image data stored in a particular way. But the DNG Converter (legitimately according to the specification) happens to use a different structure.

In other words, there is no need for a camera to implement the full generality of DNG, because it never expects to encounter it.

[stevebrot]

Adobe offered DNG to ISO years ago. ISO TC42 WG18 were expected to publish it as an ISO standard. It would then not belong to Adobe in any sense.
For some reason, (my contacts won't say why), this process appears to have stalled.

Yep...The reason for the stall is unknown, though I suspect that the lack of interoperability might be a clue.

Adobe did the same with TIFF, (which they "own"). Unfortunately ISO mangled it to become TIFF/EP, which was not prescriptive enough to be useful for interchange. It ended up more as a kit of optional parts.

The matter of TIFF is hopeless mangled from the perspective of descriptive naming and a true rat hole when trying to explain that it is more than just an image "type".

(DNG and NEF are both based on TIFF/EP. DNG was Adobe's proposal to ISO to make TIFF/EP a more useful standard).

Add PEF to your short list; it also is TIFF/EP compliant. The strange part is that DNG is a branch of sorts and not fully compliant in some of its details. Addendum...the issues with Pentax-generated DNG for in-camera HDR and Pixel-shift highlight the ambiguity. DNG technically does not support "stacked" data while TIFF/EP does.


Steve

[newmikey]

My aim isn't to change what you do. I'm trying to establish some facts that will stand up to scrutiny.

On that we agree then, but some facts will still not stand up to scrutiny.

I believe the following is true:
From when Pentax began to use DNG in cameras, they have continued to support DNG in all subsequent cameras that support raw
But the converse does not apply. There have been a number of Pentax cameras that support DNG but not PEF.
Not the major K-Mount cameras, but some nevertheless: Q-Series; K-500 (?); K-30 (?); MX-1 (?). Perhaps others.

I would say Pentax made an outing into using DNG as it standard raw format but abandoned that later on, keeping PEF as standard in all newer models (K-70, KP, K-1) with DNG for compatibility. Reading the manual for my particular model, the K-70, on page 25 it literally states under the menu listings the default format as PEF.

This defuses your statement as if "DNG is now the Pentax main and consistent raw file format, and PEF is an optional extra for historical purposes." - I'd say it is just the other way around. The reasons do not really matter to us but fact of the matter is they literally state the opposite in the camera manual so I must ask where your statement came from and what it was based upon?

I believe the DNG format has never been influenced by the needs of a Pentax camera. Those cameras have used DNG "as-is".
(I believe this leaves something to be desired when Pixel-Shift is used. But that appears to apply whether Adobe is processing a PEF or a DNG).

Wrong again. When Adobe designed DNG as a format, they actually did one thing right: they listened to camera manufacturers and left specific room for them to add their own undisclosed metadata right inside the DNG itself without the need for those manufacturers to disclose the use of that "secret sauce" to Adobe.

I accept that there is probably some non-Adobe software that supports PEFs (once they have reversed engineered them) but not DNG.

I don't really know where you get those ideas but all of my non-Adobe converters support DNG out of the box, as long as it is camera-generated DNG. It's the converted DNG's out of f.i. the Adobe DNG Converter they cannot stomach - that should tell you something about the quality of the conversion probably.

So all in all and with loads of respect for the extensive work you have done over the years (and yes, I have read it and learned from it so thanks!) there may be some bits you think are fact but have crept towards fiction over time. Maybe they were true at one stage and stuff changed. Maybe Pentax tried to go DNG-only and rethought that later on, maybe 3rd-party converters couldn't deal with DNG at first but now they can etc. etc.

As things stand today, there is no compelling reason to use DNG unless it is camera-generated and you use (and plan to continue to use into the future) Adobe software. Having said that, the once so free and happy DNG format becomes a jail for your images once it has been touched by Adobe software (either converted from proprietary raw to dng, or written to by LR) as it will take away your freedom to use non-Adobe software thereafter. There is the little thing of proof of ownership as well - a PEF file is pure raw proof of authorship as it is a true digital negative which does not get written to once it leaves the camera. DNG's might or might not be useful in proving you own a given file, depending on whether they were manipulated or converted.

So I happily use the DNG's from my Ricoh GR because they convert perfectly fine with nUfraw, DarkTable, RawTherapee or any other converter I have tried to-date, but I stick to PEF on my Pentax bodies. DNG might indeed have been a great idea and is still quite useful to many, but it is certainly far from the universal and future-proof format many people are convinced it is.

[stevebrot]

As things stand today, there is no compelling reason to use DNG unless it is camera-generated and you use (and plan to continue to use into the future) Adobe software.

That statement is an entry into one of the most glaring DNG issues. One of the strong points of DNG is that it provides for an embedded checksum for file validation. The gotcha is that the feature is only supported for Adobe-generated DNG using Adobe products.

As for Pentax RAW support:

• PEF is legacy and has been consistently offered on all higher-end bodies and most consumer-level bodies from the *istD to the present
• DNG support began with the K10D and has been available on all Pentax dSLR models since 2006
• PEF support is missing (DNG only) on some consumer-level bodies (K-30/K-50/K-500)
• PEF is has always been the default over DNG when both formats are supported

A good (if somewhat biased) discussion of DNG as part of a best practice work flow (including file validation) is discussed on the ASMP best practice Web site:

https://www.dpbestflow.org/DNG


Steve

(...shoots DNG mostly for convenience with Adobe products...never does Adobe-generated DNG...)

[stevebrot]

...and you use (and plan to continue to use into the future) Adobe software.

I figured it that last required a separate comment. I use Lightroom for my RAW conversion and have the tool configured to write XMP metadata to the library rather to the DNG. In other words, the DNG remains "virgin" with the penalty that processing metadata is never stored with the image data and might be potentially "lost", not that it really matters since the processing metadata is Adobe-specific and not readily transferable. If I were shooting PEF, I would have the option of writing the processing metadata to a "sidecar" file. I have yet to figure out an advantage there except that it is easy to find.

I never import as DNG (even as a wrapper) nor have I ever exported to DNG. I don't see the utility.

Am I bound at the hip to Adobe products (oh! the horror!)? In some ways, yes, but no more so than to any other editor or RAW processor. I continue to use Lightroom because of the full range of that tool's features (am addicted to its uncannily accurate soft-proofing feature as well as the content management piece), though truth be known, I loath Adobes business practices.


Steve

[newmikey]

Am I bound at the hip to Adobe products (oh! the horror!)? In some ways, yes, but no more so than to any other editor or RAW processor. I continue to use Lightroom because of the full range of that tool's features (am addicted to its uncannily accurate soft-proofing feature as well as the content management piece), though truth be known, I loath Adobes business practices.

Well, I suppose my view is slightly different because I value the freedom to switch raw converters once in a while or use a number of different ones in parallel. Sometimes that is because a more capable one comes along or even because one raw converter does a slightly better job on a specific type of shot under particular shooting conditions.

There's also the issue of "everything but the kitchen sink" or trying to do the whole image processing pipeline from beginning to end in one particular piece of software. I happen to like having access to different bits and pieces for different purposes so as to finetune the tool to whatever needs to be done. I am also a strong believer in "squeezing" a raw file to the utmost and being able to use different demosaicing methodologies for different circumstances. As an example I'll use AMaZE or EAHD on files with lots of small details but I'll use VNG on high-ISO files because it deals better with noise even before the image processing stage.

Lastly, I'm a strong believer in the power of open software to allow new ideas and/or updates to flow through the pipeline and make it to the user faster and at less cost. In very specific circumstances I will even tweak something myself, even though I'm not a programmer by any means. In simple terms, that allows me to tweak exif- or lensdata-library sources to correctly identify lenses I use but which are not picked up properly - a common issue on some older Tamron lenses or (the opposite) very new lenses for which the lenscorrection data is still not included in the standard libraries. (an example would be the Tammy 17-50mm/f2.8 which was always incorrectly identified as a Sigma 28-70mm by most software, or the Sigma 8-16mm but even the DA40mm/f2.8 XS).

All of the above just to explain why I need my raw file to be EXACTLY like it was when it was first recorded, without any change, and why I will always want to avail myself of whatever interesting and quirky bit of software becomes available at any given time in the future. I do not ever want to worry about compatibility which, quite controversially was the very reason many people give behind the usage of DNG. The ONLY way to absolutely guaranty that into the foreseeable future is to use PEF and avoid DNG like the plague.

[BigMackCam]

I do not ever want to worry about compatibility which, quite controversially was the very reason many people give behind the usage of DNG. The ONLY way to absolutely guaranty that into the foreseeable future is to use PEF and avoid DNG like the plague.

Interestingly, with the exception of Adobe Lightroom (and, of course, Camera Raw), most raw processing software will not load the native .ARW files from my Sony-A99-based Hasselblad HV. But if I run them through AdobeDNGConverter, a range of different software will open and process them without difficulty - including both digiKam and Darktable (which form the core of my current workflow).

I mention this only as a curiosity rather than my belief that DNG is somehow better. For this camera, DNG (or rather, lower case ".dng") is my only viable option. I suspect that AdobeDNGConverter is either removing or normalising something in the Hasselblad files to make them look the same as the A99, but that's only a guess...