[newmikey]

I'm sure anyone can come up with examples showing almost any argument but just now I went back to two raw (PEF) files I shot with the K110D in August of 2007 (!!) and I converted them with Darktable followed by a bit of levels/curves and saturation tweaking in Gimp. I should do this far more often as they come out so much better now than they did back then. Have my editing skills grown since? Sure, of course but the way DT treats these raws is an immense improvement over what I used at the time (I think it was UFRaw on Linux and the Pentax software on Windows). That's 11 years ago and I would venture to say I'm still not afraid to show these results today.

Attached Images

View Picture EXIF

File Date

 

View Picture EXIF

File Date

 

[BigMackCam]

I'm sure anyone can come up with examples showing almost any argument but just now I went back to two raw (PEF) files I shot with the K110D in August of 2007 (!!) and I converted them with Darktable followed by a bit of levels/curves and saturation tweaking in Gimp. I should do this far more often as they come out so much better now than they did back then. Have my editing skills grown since? Sure, of course but the way DT treats these raws is an immense improvement over what I used at the time (I think it was UFRaw on Linux and the Pentax software on Windows). That's 11 years ago and I would venture to say I'm still not afraid to show these results today.

Those are great results.

And just to clarify, I'm not trying to come up with examples for an argument, nor am I trying to shoot down your reasoning I'm just stating that, sometimes, the native raw format isn't supported when DNG is.

EDIT: Another example; Darktable won't open my Samsung GX-1L PEF files, telling me they're in an unknown format. So I run those through AdobeDNGConverter and it reads the .dng files just fine.

Furthermore, my previous experience with Lightroom has suggested that there is no difference in image quality (visible to me, at least) between PEF and DNG files generated by the camera, nor ARW and DNG files for my A7 MkII and Hasselblad HV.

Again, I'm only mentioning this for information and balance

Interestingly, the developers of Darktable seem to favour DNG as a "lowest common denominator" of sorts. According to the camera support information HERE, "Cameras that produce DNG files should be supported even if they are not on the list but samples are still appreciated".

[newmikey]

Interestingly, with the exception of Adobe Lightroom (and, of course, Camera Raw), most raw processing software will not load the native .ARW files from my Sony-A99-based Hasselblad HV. But if I run them through AdobeDNGConverter, a range of different software will open and process them without difficulty - including both digiKam and Darktable (which form the core of my current workflow).

I mention this only as a curiosity rather than my belief that DNG is somehow better. For this camera, DNG (or rather, lower case ".dng") is my only viable option. I suspect that AdobeDNGConverter is either removing or normalising something in the Hasselblad files to make them look the same as the A99, but that's only a guess...

RawTherapee seems to open the A99 native *.ARW files (haven tested fully but just downloaded a sample) - this maybe due to the fact RT uses a different raw decoding library than Digikam or Darktable which both seem to use LibRaw as backend. But of course, in your case converting to DNG just might make life a whole lot easier. Contrary to popular belief, one size never fits all. For you DNG might prove to be a godsend.

[BigMackCam]

RawTherapee seems to open the A99 native *.ARW files (haven tested fully but just downloaded a sample) - this maybe due to the fact RT uses a different raw decoding library than Digikam or Darktable which both seem to use LibRaw as backend. But of course, in your case converting to DNG just might make life a whole lot easier. Contrary to popular belief, one size never fits all. For you DNG might prove to be a godsend.

Yes, RT will actually open my Hasselblad HV ARW files too... but colour / white balance was way off, as I recall. It got a lot further than Darktable, though

Honestly, the HV is an oddity. For all it's truly just an A99 internally (and very similar externally, just using better materials and some slight design changes), Hasselblad chose to mess with the EXIF enough to confuse many applications. The files wouldn't load into DxO, AfterShot and (if I remember correctly) Capture One...

That aside, I agree - one size never fits all

[stevebrot]

But if I run them through AdobeDNGConverter, a range of different software will open and process them without difficulty - including both digiKam and Darktable (which form the core of my current workflow).

Well, that is something new. I have never used the DNGConverter, only Lightroom's import/export as DNG. Will it do the same with PEF?


Steve

[Dartmoor Dave]

I have never used the DNGConverter, only Lightroom's import/export as DNG. Will it do the same with PEF?

I've used the DNG converter to convert raw files from 6MP cameras that only offer PEF into DNG files to create custom profiles using Adobe Profile Editor. They work fine for that purpose, and the resulting profiles work directly on PEF files in Camera Raw without any need to do an intermediate PEF to DNG conversion for each shot. As you'd expect them to, since of course PEF and DNG are just the same candy in different wrappers.

[Just1MoreDave]

I think the OP can run K20D-generated DNG files through the Adobe DNG Converter and output lossless-compressed DNG files. Those DNGs are about the same size as a K20D PEF. As mentioned here, the Adobe DNG Converter-generated files won't open in the camera or Pentax software any more.

[newmikey]

I think the OP can run K20D-generated DNG files through the Adobe DNG Converter and output lossless-compressed DNG files. Those DNGs are about the same size as a K20D PEF. As mentioned here, the Adobe DNG Converter-generated files won't open in the camera or Pentax software any more.

Nor will they open in most any other 3rd-party converter. Seems a high price to pay for shaving off a few MB's of storage?

[photoptimist]

I think the OP can run K20D-generated DNG files through the Adobe DNG Converter and output lossless-compressed DNG files. Those DNGs are about the same size as a K20D PEF. As mentioned here, the Adobe DNG Converter-generated files won't open in the camera or Pentax software any more.

I wonder if it's a case of "the software can't open" or "the software won't open" the files. That is, the software looks at the file, sees it came from somewhere else and does not even attempt to open the file.

I once had the idea of creating some JPG maps of my destination, loading them into a suitably named folder on an SD card so that I could view the map on my K-1 whilst walking about. But the K-1 refused to open these JPGs (which had been created using MacOS Preview).

It's not uncommon for programmers to take a conservative approach and only open files from known-good sources. From a cost-of-business and software reliability perspective, it's a lot easier say program X is not compatible with camera Y than to let program X attempt open the camera Y file and then crash or do weird things that create costly customer support issues.

[Barry Pearson]

I would say Pentax made an outing into using DNG as it standard raw format but abandoned that later on, keeping PEF as standard in all newer models (K-70, KP, K-1) with DNG for compatibility. Reading the manual for my particular model, the K-70, on page 25 it literally states under the menu listings the default format as PEF.

This defuses your statement as if "DNG is now the Pentax main and consistent raw file format, and PEF is an optional extra for historical purposes." - I'd say it is just the other way around. The reasons do not really matter to us but fact of the matter is they literally state the opposite in the camera manual so I must ask where your statement came from and what it was based upon?

Having worked as an engineer in the IT industry, I beleieve there is a hierarchy of technical decision-making in Pentax:

Higher-up, there is a decision about what raw file formats are available for Pentax cameras to use. For a long time this was something like "always use DNG, optionally use PEF". We can deduce that from the raw file formats that new Pentax cameras actually used over many years. What we can't deduce is whether that decision has now changed. The last DNG-only camera I know of was announced in 2014. There have only been 6 Pentax cameras announced since then).

Below that higher-level there is a specific decision made for every camera. Perhaps the higher level policy has now changed to "DNG and PEF", or perhaps it is just an accident of history that we haven't seen any more DNG-only cameras since 2014.

Then at the lowest level there are people setting the defaults in the camera. The defaults are somewhat arbitrary. When I get a new camera I go through the menu setting the camera as I want it. I don't take the defaults seriously, because they tend to be chosen by people who don't do photography the way I do. In both the K-1ii and the K-3ii there doesn't appear to be any preference in the operating manual for one or the other. I can't remember which way they were set by default, and I don't care because they are set to DNG now!

When Adobe designed DNG as a format, they actually did one thing right: they listened to camera manufacturers and left specific room for them to add their own undisclosed metadata right inside the DNG itself without the need for those manufacturers to disclose the use of that "secret sauce" to Adobe.

It wasn't camera manufacturers who put pressure on Adobe. It was users. I was a user of DNG from 10 days after it was launched at the end of September 2004. The concern of a number of us was that, while the Adobe software at the time didn't need the Makernote which the DNG Converter had used to generate the DNG Metadata, things may change in future. Perhaps a later DNG Converter could exploit more the Makernote? Perhaps software other than that from Adobe could exploit some of the "secret auce"? (The DNG Converter can read a DNG, of course, so a thought was that if the Makernote was preserved, it could be further exploited by the latest DNG Converter).

I kept my PEFs until I was confident that they had been converted properly, and especially until the Makernote was preserved as DNGPrivateData. I was able to reconvert where relevant. Then I deleted all my PEFs. From that point on, (in 2005, about 6 months or so after I started to use DNG), after checking that a batch of conversions had worked, I had the confidence to delete my PEFs. That never caused me a problem.

Which camera manufacturers would be concerned and have sufficient influence to cause Adobe to provide DNGPrivateData? Obviously not those who had their own raw file formats and their own software to process them. Canon and Nikon hadn't made a decision use DNG in-camera and supply software that could process DNG and exploit any "secret sauce" their cameras might have put there. DNG was at best irrelevant to them, and at worst an irritant to be suppressed.

Leica was the first rumoured camera maker to use DNG in-camera, and one of the first to do so. I believe they packaged Photoshop Elements with their first cameras to use DNG. And Photoshop Elements didn't exploit any "secret sauce"in DNGs. Like other Adobe software processing DNGs, it uses the published DNG metadata. So Leica had no reason to put pressure on Adobe.

Ricoh's GR series was the first compact camera to use DNG, and in fact Ricoh exploit DNG more than most. They use DNG 1.3.0.0 in order to use the lens-correction opcodes. (The K-1-series "only" uses DNG 1.2.0.0).

It is worth noting that, when I stopped counting in 2010, there were 14 cameras makers using DNG, 47 camera models using DNG, 240+ software products able to process DNG to some or a full extent, and 290+ Adobe-Convertible raw image formats. And about 7 or 8 alternative DNG converters to cater for niche or hacked cameras that the Adobe DNG Converter didn't cater for.

For comparison, during the first 5 years when about 38 camera models were launched that wrote DNG, Adobe software added support for about 21 Canon models, about 20 Nikon models, and about 22 Olympus models.

I don't really know where you get those ideas but all of my non-Adobe converters support DNG out of the box, as long as it is camera-generated DNG. It's the converted DNG's out of f.i. the Adobe DNG Converter they cannot stomach - that should tell you something about the quality of the conversion probably.

That says more about those raw converters than about DNG! DNG is comprehensive. It caters for more capability than a typical camera uses. After all, Adobe software manages with just the published DNG specification features of DNG, so it can be done.

As things stand today, there is no compelling reason to use DNG unless it is camera-generated and you use (and plan to continue to use into the future) Adobe software.

I take the converse view. PEF is irrelevant to me, so the fact that my Q-Series cameras didn't use PEF wasn't an issue to me. All I ask from a Pentax camera is "please use DNG; it doesn't matter whether you support PEF".

Having said that, the once so free and happy DNG format becomes a jail for your images once it has been touched by Adobe software (either converted from proprietary raw to dng, or written to by LR) as it will take away your freedom to use non-Adobe software thereafter.

Why?

I can take a K-1ii DNG which I've used in Lightroom, put it onto an SD Card, put it back into the K-1ii, and view it. In fact, a minute before I wrote this I did precisely this! It has not been changed by Lightroom. (I've also just used the DNG SDK to examine the DNG file and verify this).

I suspect you are talking about using Lightroom to store the catalogued metadata editting back into the DNG. Rather than say, storing it as an XMP file, or simply using the Lightroom catalogue and not interfere with the DNG. I don't do that.

There is the little thing of proof of ownership as well - a PEF file is pure raw proof of authorship as it is a true digital negative which does not get written to once it leaves the camera. DNG's might or might not be useful in proving you own a given file, depending on whether they were manipulated or converted.

I don't let my DNGs get changed by Lightroom. But PEFs can also be changed by software! To the best of my knowledge, there is no way of detecting whether a PEF has been changed since it left the camera.

---------- Post added 1st Nov 2018 at 02:41 PM ----------

Interestingly, with the exception of Adobe Lightroom (and, of course, Camera Raw), most raw processing software will not load the native .ARW files from my Sony-A99-based Hasselblad HV. But if I run them through AdobeDNGConverter, a range of different software will open and process them without difficulty - including both digiKam and Darktable (which form the core of my current workflow).

I mention this only as a curiosity rather than my belief that DNG is somehow better. For this camera, DNG (or rather, lower case ".dng") is my only viable option. I suspect that AdobeDNGConverter is either removing or normalising something in the Hasselblad files to make them look the same as the A99, but that's only a guess...

Interesting!

See my experience at the address below:
Support via DNG but not native raws

I one found that Phase One Capture One wouldn't support DNGs derived in any way from rival digital backs. So I used a hex-editor to change the camera/back name in the DNG to a same-size meaningless string of letters, and the software then worked!

It deliberately checked for, and rejected, DNGs that had the names of rivals in them!

---------- Post added 1st Nov 2018 at 02:47 PM ----------

Nor will they open in most any other 3rd-party converter. Seems a high price to pay for shaving off a few MB's of storage?

When I stopped counting in 2010, there were 14 cameras makers using DNG, 47 camera models using DNG, 240+ software products able to process DNG to some or a full extent, and 290+ Adobe-Convertible raw image formats. And about 7 or 8 alternative DNG converters to cater for niche or hacked cameras that the Adobe DNG Converter didn't cater for.

Products that support DNG in some way

[stevebrot]

I can take a K-1ii DNG which I've used in Lightroom, put it onto an SD Card, put it back into the K-1ii, and view it. In fact, a minute before I wrote this I did precisely this! It has not been changed by Lightroom. (I've also just used the DNG SDK to examine the DNG file and verify this).

Import as DNG and try the same thing. You are correct that LR will not rewrite an in-camera DNG unless instructed or configured to do so. As you mentioned, updating catalog metadata to file is a common way that happens.

I one found that Phase One Capture One wouldn't support DNGs derived in any way from rival digital backs. So I used a hex-editor to change the camera/back name in the DNG to a same-size meaningless string of letters, and the software then worked!

Ha! Ha! Take that Phase One!


Steve

[newmikey]

Having worked as an engineer in the IT industry, I beleieve there is a hierarchy of technical decision-making in Pentax:

Higher-up, there is a decision about what raw file formats are available for Pentax cameras to use. For a long time this was something like "always use DNG, optionally use PEF". We can deduce that from the raw file formats that new Pentax cameras actually used over many years. What we can't deduce is whether that decision has now changed. The last DNG-only camera I know of was announced in 2014. There have only been 6 Pentax cameras announced since then).

Below that higher-level there is a specific decision made for every camera. Perhaps the higher level policy has now changed to "DNG and PEF", or perhaps it is just an accident of history that we haven't seen any more DNG-only cameras since 2014.

It's weird how facts can be spun. I've always thought the period of DNG-only for Pentax was really only a blip in time. As you said, 2014 and that's already 4 years and 6 models ago.

Then at the lowest level there are people setting the defaults in the camera. The defaults are somewhat arbitrary. When I get a new camera I go through the menu setting the camera as I want it. I don't take the defaults seriously, because they tend to be chosen by people who don't do photography the way I do. In both the K-1ii and the K-3ii there doesn't appear to be any preference in the operating manual for one or the other. I can't remember which way they were set by default, and I don't care because they are set to DNG now!

You know, we have a different way of looking at things. I read the manual and look at the menus - if both specifically go out of their way to confirm PEF is the default format, I just assume that to be the case. Maybe my thinking is too simple and one-dimensional, I'll give you that.

I kept my PEFs until I was confident that they had been converted properly, and especially until the Makernote was preserved as DNGPrivateData. I was able to reconvert where relevant. Then I deleted all my PEFs. From that point on, (in 2005, about 6 months or so after I started to use DNG), after checking that a batch of conversions had worked, I had the confidence to delete my PEFs. That never caused me a problem.

Well, that goes a long way to explaining why you're so anxious. As you deleted all of your PEF's, you're in 100% and cannot go back.

Which camera manufacturers would be concerned and have sufficient influence to cause Adobe to provide DNGPrivateData? Obviously not those who had their own raw file formats and their own software to process them. Canon and Nikon hadn't made a decision use DNG in-camera and supply software that could process DNG and exploit any "secret sauce" their cameras might have put there. DNG was at best irrelevant to them, and at worst an irritant to be suppressed.

Leica was the first rumoured camera maker to use DNG in-camera, and one of the first to do so. I believe they packaged Photoshop Elements with their first cameras to use DNG. And Photoshop Elements didn't exploit any "secret sauce"in DNGs. Like other Adobe software processing DNGs, it uses the published DNG metadata. So Leica had no reason to put pressure on Adobe.

Well, sometimes the "secret sauce" gets reverse engineered by 3rd-parties such as the Exiv2 project. I've got nothing against DNG-only cameras, I have a Ricoh GR. I just don't see what value they add. (EDIT: I should have added "other than the camera quality itself")

I suspect you are talking about using Lightroom to store the catalogued metadata editting back into the DNG. Rather than say, storing it as an XMP file, or simply using the Lightroom catalogue and not interfere with the DNG. I don't do that.

It's not whether you do it but whether it can be done at all! You made a sensible choice but many people could easily be tricked into doing exactly what you know not to do, and that is the problem with this whole DNG-mess.

I don't let my DNGs get changed by Lightroom. But PEFs can also be changed by software! To the best of my knowledge, there is no way of detecting whether a PEF has been changed since it left the camera.[COLOR="Silver"]

I do not think I have ever come across software, commercial or open source which can change a PEF/NEC/CR2/,, or any other proprietary raw file. Anyone with a hex editor can hack any file but that is not the point. The point being that you can very easily be tricked into writing directly to a DNG file but never to a proprietary raw while using a standard raw converter.

[stevebrot]

It's not whether you do it but whether it can be done at all! You made a sensible choice but many people could easily be tricked into doing exactly what you know not to do, and that is the problem with this whole DNG-mess.

That is the problem with the Adobe-mess. Martin Evening discusses the topic in detail in his popular LR books and make a straight statement regarding whether there is any value in writing XMP to file. The use cases for the feature are rather weak.

I do not think I have ever come across software, commercial or open source which can change a PEF/NEC/CR2/,, or any other proprietary raw file.

That is true if one does not consider the EXIF (and related) metadata part of the file. I have rewritten PEF with ExifTool...no problem.


Steve

[newmikey]

That is true if one does not consider the EXIF (and related) metadata part of the file. I have rewritten PEF with ExifTool...no problem.

Yep, oversight on my side - exiftool does indeed write metadata to raw files.

[Barry Pearson]

Well, that goes a long way to explaining why you're so anxious. As you deleted all of your PEF's, you're in 100% and cannot go back..

I deleted PEFs until I started to use DNGs in-camera. Then there were no longer any PEFs to be deleted!. That happened perhaps a decade or so ago.

It should be obvious that I am not, and never have been, anxious! I have never had a reason to be anxious, once I established that PEFs were irrelevant and useless to me.