[newmikey]

I have never had a reason to be anxious, once I established that PEFs were irrelevant and useless to me.

And I think that's exactly where the problem really is. It may be irrelevant and useless to you but there are more ways than your way.

As you wrote umpteen years ago on your blog, written in 2009 I believe and last updated in 2013:
"DNG, was launched by Adobe on 27 September 2004. It will transform the world of raw shooting within years."
Well, it is 2018 now, some 14 years later. I still cannot see that "transformation" today. Most photographers on Nikon or Canon platforms use NEF and CR2 respectively, Sony users use ARW. Lots of Pentax users stick with PEF. DNG simply did not make the big splash you were expecting. Part of that is probably the lost virtual monopoly that Adobe once had and the fact there is a plethora of commercial as well as free options available in-market today. It is perfectly fine for you to only consider only yourself as you did in your previous post but you are not the yardstick by which the world is measured. (nor am I for that matter)

"It will open up raw shooting to more people, more products, and more opportunities, and make it easier for photographers and users of photographs to build more valuable and comprehensive workflows."
It actually didn't do that either. Nothing has become "easier" for photographers compared to what the situation was 14 years ago purely on account of the introduction of DNG alone. DNG turned out to be just one more raw format.

"It will become the accepted archival raw file format. DNG is "the new TIFF", at a different place in people's workflows"
This did not happen either.

[Barry Pearson]

"It will open up raw shooting to more people, more products, and more opportunities, and make it easier for photographers and users of photographs to build more valuable and comprehensive workflows."

When I stopped counting in 2010, there were 14 cameras makers using DNG, 47 camera models using DNG, 240+ software products able to process DNG to some or a full extent, and 290+ Adobe-Convertible raw image formats. And about 7 or 8 alternative DNG converters to cater for niche or hacked cameras that the Adobe DNG Converter didn't cater for.

DNG has made it easier for niche and minority camera makers to output raw files and get their raw files processed.

Suppose that you were developing a digiscope that can record raw files. You need a raw file format to write. You needed raw file converters to process those raw files. What do you do?

Answer (and it has been done): use DNG. The format is published and freely available. There is an SDK to help develop the DNG code. Once completed, there are perhaps 100 or more software products that can now process your images.

There are many other niche and minority cameras that have benefited in the same way. Life became easier for those camera makers and users.

DNG turned out to be just one more raw format.

DNG is unlike any other raw file format.It is the only one with a freely available specification. It is the only one with a freely available (no questions asked and no NDA) SDK comprising C++ and executables. There are various other aids for using it too:
Digital Negative (DNG), Adobe DNG Converter | Adobe Photoshop CC

There is a weird attitude that some people have towards DNG. It is fairly common for people to say "DNG has problem X, so I'll continue using my ... [CR2s] ... [NEFs] ... [PEFs] ... ". Yet those raw file formats not only also suffer from problem X, but also lots more problems. Somehow, their anti-DNG (or perhaps anti-Adobe) attitude is so strong that they think a problem X with DNG (even if it exists) is vastly more serious than the same problem X and lots more with their own camera makers' non-DNG raw files.

Another common trap that people fall into is that because the format of DNG is specified they can see things that they think (rightly or wrongly) are problems. But because the format of their own camera makers' non-DNG raw files are not published, they can't see what problems are lurking there, so they appear to assume there aren't any!

Some people have criticised DNG for allowing data of unpublished format to be output as DNGPrivateData in the file. While ignoring the fact that everything in their own camera makers' non-DNG raw files is in unpublished format! It is hard to find faults with DNG that are not present (along with many more) in camera makers non-DNG raw files.

DNG is the only archival raw file format

Endorsements for DNG

How many other raw file formats are used by cameras of several different manufacturers? And why not? (As far as I know, DNG is the only raw file format for which explicit permission has been published for everyone and all organisations to use it).

[newmikey]

Barry, you're a hard-headed guy and I respect that. You make some valid points and I respect those too. But what you fail to see is the reverse of what you state above: It is hard to find faults with PEF that are not present (along with many more) in Pentax's implementation of DNG and there are more than enough faults with a DNG which are the result of conversion with Adobe DNG Converter (or any 3rd-party DNG converters for that matter).

That being the case, the whole reasoning for Pentax owners of most models to use DNG rather than PEF does not exist. You are an engineer, a mathematician - you cannot NOT see the logic of that. The statement "DNG has problem X, so I'll continue using my PEFs with the same problem X" is as invalid as "PEF has problem X, so I'll start using DNGs with the same problem X".

You have made this DNG thing into a religion of sorts and both you and I agree on one thing - we do not care for such a belief-based system!

[BigMackCam]

Fellas - I sense some emotion in this thread, so please let's keep it cordial, eh? We're all entitled to our opinions, as I think everyone posting in the thread respects - and it's been friendly so far. Thanks in advance

Out of moderator mode...

I'm finding this educational, especially considering the choices I've made, and have had to make, thus far

[newmikey]

Fellas - I sense some emotion in this thread, so please let's keep it cordial, eh? We're all entitled to our opinions, as I think everyone posting in the thread respects - and it's been friendly so far. Thanks in advance

I thought I kept it pretty clean but if you prefer, I'll take a break here. Barry and I will probably never agree on the DNG thing anyway although I have read through his blogs and website and we do have a lot in common in non-photography related stuff, I can assure you! So if you read emotion I'll readily apologize - I just love a good wordfight that's all and Barry is a good fighter.

[BigMackCam]

I thought I kept it pretty clean but if you prefer, I'll take a break here. Barry and I will probably never agree on the DNG thing anyway although I have read through his blogs and website and we do have a lot in common in non-photography related stuff, I can assure you! So if you read emotion I'll readily apologize - I just love a good wordfight that's all and Barry is a good fighter.

No apology required, Mike Everyone's been cordial, but whether correctly or incorrectly, I sensed some tension. With less balanced folks than yourselves, that can sometimes turn unpleasant. I'd prefer you stick with the thread... You and Barry are both sources of good info (even if some of it is in disagreement) for me and a bunch of others here

[photoptimist]

Personally, I see both sides of this argument.

The deeper issue with DNG vs. PEF, NEF , CR2, etc. is the near impossibility of creating a true interoperable standard in an industry driven by rapid innovation. Although, in theory, the basics of digitized data from a pixel array light sensor are simple enough to be encoded in a standard, there's enough change in the secret sauce to imply that new cameras always need some new code. DNG offers less than one might hope because new cameras always mean new file types even if the three-letter file extension is an old standard one.

But the issue of whether old files from old cameras remain accessible is a trickier problem. Just because somebody wrote some code for opening old PEF files once, does not mean that this code will be reusable in perpetuity. The original PEF code may require rewriting as operating systems, APIs, and languages evolve. Obsolesce may be only a Window's update away. And if new application developers arise, they are not likely to put much effort into supporting ancient camera models made by ancient camera brands. Here, the chance of maintaining access to an old file is higher if is in a more popular format that is still be used by surviving camera makers. I'd wager (but not anything of value) that there are more DNG files being created per day than PEF files. The chance of being able to open an old DNG file would be higher than the chance of being able to open an old PEF file. And if the file cannot be opened because the software does not support some ancient camera model, then the opportunity to hack the file to make it open would be better with DNG than with PEF.

Overall, I personally use DNG because it seems more widely compatible and more likely to be readable 10, 20, 30 years down the road.

[newmikey]

But the issue of whether old files from old cameras remain accessible is a trickier problem. Just because somebody wrote some code for opening old PEF files once, does not mean that this code will be reusable in perpetuity. The original PEF code may require rewriting as operating systems, APIs, and languages evolve. Obsolesce may be only a Window's update away.

But support for PEF/NEF/CR* ec is not linked to the operating system to begin with. Support for existing proprietary raw formats is always available in its most basic form and free for anyone to (re)implement through existing C libraries. That language is not really going anywhere soon and even if it were, the structure is so common it can be ported to any newer language which might come around the corner.

The argument of obsolescence has been used many times but the truth is that most raw converters and libraries today read raw formats harking back to the Sony Mavica models (yes, the ones with the floppy disk and 0.9Mp). Truth be told, my Darktable 2.4.4 still reads files from my Casio QV4000 I purchased in 2013 as well as every single Canon Powershot and Pentax camera I owned over the last 15 years - none excluded. My software toolkit evolved from plain DCRAW into UFraw, nUfraw, Rawtherapee, Bibble, Digikam and now Darktable and not one single historic format ever got mislaid in the process.

With Microsoft no longer myopic and now contributing to- and absorbing from- the open source community one could use the exact same argument which is used to promote the DNG format: the ability to read existing (even historic) proprietary raw formats is openly published, for anyone to implement at any given moment. DNG certainly is no less future-proof than proprietary raw formats but no more either. In that, it really is "yet another raw format".

Chances are both Windows, IOS as well as Linux will be long gone but raw formats both proprietary formats as well as DNG will remain readable by anything new that comes along.

If we are to be afraid of obsolescence, it should be the storage media we must fear. Just like it is almost impossible to obtain a 5.25 floppy drive today, CD/DVD drives are quickly disappearing from computers, USB may go the way of the dodo as may cloud storage. We may be using atom crystalline structures or some form of optical holographics in future. Even if our raw formats will all be readable 20 years into the future, the storage media we maintain the files on may become obsolete way before that. DNG or PEF will not matter one bit if you lack the means to actually read the files off old-fashioned USB3 SSD drives.

Possibly the only thing to really survive longer will be archival-quality prints (unless moisture, acid rain or beasties eat away at them). In the end, nothing will remain but for the foreseeable future the format should be the least of our worries.

[Barry Pearson]

You have made this DNG thing into a religion of sorts and both you and I agree on one thing - we do not care for such a belief-based system!

This isn't about me! It is about comparing PEF with DNG.

We need a link to the specification of PEF so that we can see if it has technical advantages over DNG. (Anyone?)

We need a link to the PEF SDK so that we can examine whether PEF is easier to develop for than DNG. (Anyone?)

Are there any cases where where other camera manufacturers felt that it was better to use PEF than DNG? (Several have used DNG).

Why doesn't Ricoh use PEF for Ricoh-branded cameras? (They use DNG).

Why didn't Pentax feel the need to support PEF in the four Q-System cameras? (They used DNG).

We need links to any cases where PEF had been recommended or endorsed as an archival raw file format. (DNG has).

Why did the US Library of Congress endorsed the use of DNG rather than PEF for sustainability of their photography collections?
Re: DNG - yes or no?: Nikon FX SLR (DF, D1-D5, D600-D850) Talk Forum: Digital Photography Review

Here is a 2005 discussion about this topic:
PhotoshopNews: Photoshop News and Information » Archive » Digital Preservation

Here is a 2011 discussion about DNG and Digital Access Management:
Digital Asset Management Meets Adobe DNG

As far as I can tell from everything written in this thread, the primary merit that has been identified for PEF compared with DNG is that some people simply prefer using PEF!

[stevebrot]

Why did the US Library of Congress endorsed the use of DNG rather than PEF for sustainability of their photography collections?
Re: DNG - yes or no?: Nikon FX SLR (DF, D1-D5, D600-D850) Talk Forum: Digital Photography Review

My understanding is that archivists are still in the TIFF camp in that RAW captures have no interpretation applied. I.e. the value of DNG is for the photographer not the archivist.

FWIW, I am agnostic regarding DNG vs. PEF. They are data equivalent with the only strong practical case for in-camera DNG being that Adobe products provide support using the embedded dcp profile, regardless of camera model. My output (JPEG, processed RAW, processed negative scans, digital prints, and silver prints) are all process-bound with the source capture (file or negative) being of long-term interest only to art historians and then only if I left copious notes.

Cross-tool support is a minor advantage to me once I have done the PP work for a particular interpretation (I often do several per capture). Alternative tools have no reference to what I have done in LR and even with some form of XMP migration, the results would not be fully equivalent.


Steve

(...slave to my Adobe overlords...)

[Barry Pearson]

If we are to be afraid of obsolescence, it should be the storage media we must fear. Just like it is almost impossible to obtain a 5.25 floppy drive today, CD/DVD drives are quickly disappearing from computers, USB may go the way of the dodo as may cloud storage. We may be using atom crystalline structures or some form of optical holographics in future. Even if our raw formats will all be readable 20 years into the future, the storage media we maintain the files on may become obsolete way before that. DNG or PEF will not matter one bit if you lack the means to actually read the files off old-fashioned USB3 SSD drives.

Here is a possibility that I haven't analysed much yet: The Cloud.

I've experimented with the new cloud-based Lightroom CC. (Which is different from the desktop-based Lightroom Classic CC, which is a descendent of the familiar Lightroom 6).

As I added more files on one PC, they ended up on the Adobe Cloud. They were then accessible from my other PC, and from a web browser after login at lightroom.adobe.com. I think they were also accessible from an app on Android or iOS. (I was using DNG, but I assume this would apply to other raw formats, and it certainly works with JPEG).

They were stored on media maintained by Adobe; both the raw files and the editing metadata. (There is some tricky synchronisation involved!) So the storage medium is the responsibility of a big corporation rather than individual users. (That itself raises different questions, of course).

While the photos are on the Adobe Cloud, various added-value operations can be performed on them, such as keywording (and I think facial recognition). I assume both archiving and storage media migration will routinely be performed.

Somewhere in all of that are clues about how future-proofing might happen.

[newmikey]

Somewhere in all of that are clues about how future-proofing might happen.

If the experience of the past 10 years have taught me anything, it would be that cloud-storage by a 3rd-party can be unplugged at a moment's notice and that companies sometimes take a dramatic turn away from the services or products they have been offering for many years.

The Fotki debacle is already a couple of years ago but anyone foolish enough to have taken up the offer of cloud-based storage from them woke up one day to find themselves locked out of their own files unless they paid what I can only describe as a ransom. Very fresh is this week's news about Flickr of course.

But do not forget the hardware side of things with established names divesting of whole divisions in a matter of months - in computing, IBM springs to mind of course and in industrial areas GE, one of the longest existing industrial concerns, has been shedding complete businesses left, right and center (I used to work for GE so I keep up with the news through friends who are still there as well as the online feeds. Adobe could be doing industrial 3D printing 3 years from now for all we know.

No, you simply do not trust cloud-based storage for anything else but convenience services. Even cloud service still relies om Adobe or another party having dedicated servers with whatever the current storage technology happens to be. It's not as if the word "cloud" has suddenly started to mean "no hardware involved".

It does sound very neat and convenient that you can access all of your stuff via the Adobe cloud but I can do the exact same thing with my own NAS at home without being locked-in to one provider or commercial party. (actually that's exactly what I do with triple redundancy drives one of which is stored outside the home)

So it's all very neat that the same company which brought us "the best archival format" also brings us "the best cloud-based archive" but that is a lot of eggs for anyone to put in one single basket!

Steve made an excellent point above I have to admit to not having thought about where he stated:

My output (JPEG, processed RAW, processed negative scans, digital prints, and silver prints) are all process-bound with the source capture (file or negative) being of long-term interest only to art historians and then only if I left copious notes.

And that puts the question to all of us whether raws, PEF/NEF/CR* or DNG actually are what we want long-term archival to look like? Would I want some future person to take a raw I shot and process it to his/her own liking without knowing how I envisaged my work? Possibly for straight documentary shots the answer might be "sure, why not?" but for the more creative work I suppose a 16-bit TIFF would work way better as an archive of what I made/created.

Some excellent thought processes in this thread indeed!

[Barry Pearson]

My understanding is that archivists are still in the TIFF camp in that RAW captures have no interpretation applied. I.e. the value of DNG is for the photographer not the archivist.

There are two separate aims, depending on the purpose of the archiving:

First, to archive the scene. This would be of use to future historians. This is rather like "photojournalism" or "wildlife" in a competition, where the winning photographer may be required to supply the original raw file so that judges can ensure that the photo is a "true representation". DNG works for this case.

Second, to archive the photographer's interpretation. This is of use to future curators of photography. The archive needs to be as close as possible to the photographer's chosen output medium. TIFF works for some cases, but in other cases prints need to be preserved.

---------- Post added 3rd Nov 2018 at 07:24 AM ----------

And that puts the question to all of us whether raws, PEF/NEF/CR* or DNG actually are what we want long-term archival to look like? Would I want some future person to take a raw I shot and process it to his/her own liking without knowing how I envisaged my work? Possibly for straight documentary shots the answer might be "sure, why not?" but for the more creative work I suppose a 16-bit TIFF would work way better as an archive of what I made/created.

I think "Possibly for straight documentary shots" corresponds to part of my response to Steve.