Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
11-06-2013, 08:14 PM   #1
Forum Member




Join Date: Aug 2012
Photos: Albums
Posts: 50
DNG back to RAW conversion?

Is there any way to convert the Pentax DNG format from K-30 back to a RAW format (PEF, ...)? Note that only DNG and/or JPG and no PEF can be be created in the K-30!
The reason for this strange request is, that I dislike modifications of my image files from software like Lightroom or other tools and I prefer XMP sidecar files.
This has also big benefits for backup speed.

Setting read-only attributes for DNG files is no solution, because depending on program settings, Lightroom and other DAM tools still want to update DNG with metadata or updated previews. If DNG files are read-only error messages or blocked tools (e.g. through infinitive ExifTool loops) can result.

I tried to rename DNG file extension to something else (e.g. *.rah or *.pex). It is interesting, Lightroom v5.2 still identifies DNG and is able to edit the files. But sadly Lightroom will still read and embed the XMP date in such files. Other DAM tools will not read such files at all and some others tools (e.g. IMatch) will read and write sidecars and even can display the preview of such files correctly.

Peter

11-06-2013, 09:04 PM - 1 Like   #2
Veteran Member
VisualDarkness's Avatar

Join Date: Nov 2011
Location: Uppsala, Sweden
Photos: Gallery
Posts: 4,441
DNG is a RAW-file.
11-07-2013, 03:16 AM   #3
Forum Member




Join Date: Aug 2012
Photos: Albums
Posts: 50
Original Poster
Of course DNG files are RAW files. My wording was perhaps not good enough, but you hopefully understand my wish to protect DNG files from being edited.
I am a non native english speaker/writer.

Shall I name a proprietary raw file from camera PRAW. But what is then a DNG file with embedded PRAW compared to a DNG file without embedded PRAW?
And finally the embedded image information in DNG can be still camera manufacturer proprietary and without special knowledge a tool cannot display and process the raw content correctly. This is still the case with version 6.x of CaptureOne. And in C1 version 7.x the K-30 is not mentioned as supported (but there are romours that this should be the case).

Let's try it again. To my knowledge there are a couple of different types of RAW files:
  1. original proprietary RAW files directly from camera with file extensions *.CR2 *.RAW *.IIQ *.PEF ...
  2. original DNG file from camera
  3. converted DNG files from 1. without the original raw data
  4. converted DNG files from 1. including the original raw data
  5. converted DNG files 2.-4. with modified preview (no, medium, large)
  6. proprietary RAW file containers (e.g. ZIP files) with file extensions like *.EIP
  7. modified DNG files from 1.-5. which include proprietary modifications stored by RAW editor or text metadata modified by DAM software
I like to protect completely my original files from camera (both 1. and 2.) and like to avoid case 7. completely!
This is easily possible for files 1. but apparently not for files 2.
Write protected DNG files are not supported from a lot of tools including Lightroom, when you prefer XMP sidecars. By the way very similar to JPG files.

Peter

Last edited by Plentax; 11-07-2013 at 09:47 AM.
11-07-2013, 11:52 AM   #4
Loyal Site Supporter
Canada_Rockies's Avatar

Join Date: Nov 2007
Location: Sparwood, BC, Canada
Photos: Gallery | Albums
Posts: 9,447
Lightroom does not modify the original file. It merely stores a script containing the changes you want before exporting the file.

11-08-2013, 02:55 AM   #5
Forum Member




Join Date: Aug 2012
Photos: Albums
Posts: 50
Original Poster
QuoteOriginally posted by Canada_Rockies Quote
Lightroom does not modify the original file. It merely stores a script containing the changes you want before exporting the file.
That is not fully correct. For JPEG, TIFF, PNG, PSD and DNG files, Lightroom can/will save metadata directly in the files.
Of course LR does not modify the original image content, but the embedded XMP data is modified.
For all other supported files metadata will be written to sidecar XMP files next to the original photo.
With DNG files there is also the option to update modified preview and metadata in the file.

If a user wants to keep JPGEG, TIFF, ... untouched (not modified) there is no real chance with LR.
Write protection does not help, because LR reports an error in this case.
The requriement to use XMP sidecars can have multiple reasons:
  1. Backup process and speed
  2. Protecting original files
  3. Principle
  4. Avoid placement of proprietary data in original file
Regarding point 4, all LR edit data is proprietary and cannot be used by other software including some other Adobe software.
Why should I "pollute" my valuable original files with proprietary data and make the backup process too complicated and with overhead.

I like to use XMP sidecars for ALL my original media files. With DNG and LR this looks impossible.
Other tools like GeoSetter, JPhotoTagger, Imatch, Idimager, PhotoSupreme, ... can respect this requirement.

In another forum I read about the proposal to keep always two DNG files: One original DNG and a working DNG copy for editing.
This workaround is of course feasible, but is a big waste of storage and still complicating the regular backup process.

Peter
11-08-2013, 03:39 AM   #6
Pentaxian
amoringello's Avatar

Join Date: May 2008
Location: Virginia, USA
Photos: Gallery
Posts: 1,427
Sadly, RAW formats are proprietary and by definition there is no way to reliably and legally re-generate files in those formats.

It sounds a little bit like a case of OCD (Obsessive Compulsive Disorder) to worry about such things.

I used to import to Lightroom as DNG with a second copy being sent to my backup drive. (DNGs are better compressed and more efficiently handled by Lightroom than PEFs)
Overnight all those files get backed up to offline storage (Backblaze - like Carbonite).
Any edits I make during the day are protected by the second untouched backup made during import.
Backups run at night and overwrite those, but the Offline storage now has a history. If anything should go wrong, I have at least six months of history from which to recover.
Beyond that, I have never had an issue with actually needing the clean original file, and I finally stopped making the second backup on import.

I have more often lost or separated the XMP files.
The amount of added disk space and backup times for those "small" files is enormous (probably nearly double the XMP file size).
The header space on disk is often larger than the small files themselves. Time to create many small files during backup is much slower than fewer large files. (although incremental backups could be much faster with mass edits across many files)
It all became a hassle I didn't want to deal with.

But if you have a good error-free workflow then stick with it!!
Sadly, all you can do now is kept a clean copy of the original DNG separated so they don't get filled with modified metadata.
11-08-2013, 04:21 AM   #7
Forum Member




Join Date: Aug 2012
Photos: Albums
Posts: 50
Original Poster
Whaoo, that was a profound answer from amoringello. Good, helpfull, not only black&white. Thank you for sharing your view and experience.

I can confirm that backup of multiple small modified XMPs can be time consuming depending on backup tools and strategy.
But the sidecars do not contain very important information and can be regenerated quite easily from the DAM and Lightroom databases.
These databases are the core assets, which need consequent backup strategy. The XMP sidecars are just partial exports from them.
That is why I backup regularly all media files, but do not care so much about the XMP sidecars.
The sidecars can very helpful in the case of migration between different tools,
because a direct meta data transfer between databases is normally not possible.

The key issue behind is, that both: 1. the XMP sidecar concept or 2. the XMP embedded concept
cannot be realized completely for all file types/extensions from most DAM and image editor tools.
Case 1 is not supported from most tools and case 2 is only possible for a few file types.
Everybody has to find his individual workflow and tools with this metadata discrepancy.

Peter

Last edited by Plentax; 11-08-2013 at 04:27 AM.
11-08-2013, 06:05 AM   #8
Pentaxian




Join Date: Sep 2012
Photos: Gallery | Albums
Posts: 1,728
QuoteOriginally posted by Plentax Quote
Whaoo, that was a profound answer from amoringello. Good, helpfull, not only black&white. Thank you for sharing your view and experience.

I can confirm that backup of multiple small modified XMPs can be time consuming depending on backup tools and strategy.
But the sidecars do not contain very important information and can be regenerated quite easily from the DAM and Lightroom databases.
These databases are the core assets, which need consequent backup strategy. The XMP sidecars are just partial exports from them.
That is why I backup regularly all media files, but do not care so much about the XMP sidecars.
The sidecars can very helpful in the case of migration between different tools,
because a direct meta data transfer between databases is normally not possible.

The key issue behind is, that both: 1. the XMP sidecar concept or 2. the XMP embedded concept
cannot be realized completely for all file types/extensions from most DAM and image editor tools.
Case 1 is not supported from most tools and case 2 is only possible for a few file types.
Everybody has to find his individual workflow and tools with this metadata discrepancy.

Peter
I'm not sure I understand the problem. Isn't it a core feature of the DNG file format that the original sensor output will always remain readable by any software application that can read DNG files?

11-08-2013, 08:41 AM   #9
Loyal Site Supporter
Canada_Rockies's Avatar

Join Date: Nov 2007
Location: Sparwood, BC, Canada
Photos: Gallery | Albums
Posts: 9,447
LR does not modify the original sensor output, but apparently from the posts above, it does in fact modify the EXIF data in the file. I was not aware of this, and am not personally concerned with this particular situation. As long as I do not lose any image data, I am happy. It appears that others are not, which is of course their privilege.
11-08-2013, 09:01 AM   #10
Senior Moderator
Loyal Site Supporter
Parallax's Avatar

Join Date: Apr 2008
Location: South Dakota
Photos: Gallery
Posts: 15,591
There is an option under "Catalog Settings" to have LR automatically write changes to xmp files. It doesn't specifically say if that prevents it from writing it to the DNG, but there is a warning under that option that says that changes made in LR will not automatically be visible in other applications.
11-08-2013, 05:01 PM   #11
Loyal Site Supporter
Canada_Rockies's Avatar

Join Date: Nov 2007
Location: Sparwood, BC, Canada
Photos: Gallery | Albums
Posts: 9,447
That's a strong hint that it writes to XML rather than EXIF, Parallax. I think I'll check with Adobe user group to see if that is true.
11-11-2013, 08:56 AM   #12
Forum Member




Join Date: Aug 2012
Photos: Albums
Posts: 50
Original Poster
QuoteOriginally posted by IchabodCrane Quote
I'm not sure I understand the problem. Isn't it a core feature of the DNG file format that the original sensor output will always remain readable by any software application that can read DNG files?
This is another issue. YES the DNG data can be read by most RAW software, but NO the result may not be as expected. There are a couple of issue:
  1. Some free software included with digital cameras does read only specific DNG files from this camera manufacturer (e.g. SilkyPix).
    In order to read DNG from other cameras the user has to register and pay this software.
  2. Some software will not open DNG files from new camera models which are not yet supported.
  3. Some software cannot interpret the sensor data correctly, because the camera model is not yet supported. This can result in very different issues, e.g. brighness or color deviations. I experienced this for K-30 DNG files with CaptureOne v6.4. Only C1 v7.x seems to be able to read these DNG files correctly.
Important is, that you carefully check, that a software is really using the raw data in the DNG and not just displaying (or even processing) the embedded JPG preview. In order to see the difference I am replacing the full resolution preview in the DNG files from my K-30 allways with a medium preview. With this you can see immediately what a software is reallly doing. And you can reduce DNG file size a little bit.

So the fact is, that DNG files can contain proprietary camera data, which needs correct interpretation. This is a specific engineering effort, which requires time and communication with camera manufacturer. To my humble understanding DNG is just a standardized container (like ZIP file), which can contain proprietary raw content and non proprietary meta-text information.

Peter

Last edited by Plentax; 11-11-2013 at 09:02 AM.
11-11-2013, 11:16 AM   #13
Pentaxian
amoringello's Avatar

Join Date: May 2008
Location: Virginia, USA
Photos: Gallery
Posts: 1,427
QuoteOriginally posted by Plentax Quote
This is another issue. YES the DNG data can be read by most RAW software, but NO the result may not be as expected. There are a couple of issue:
  1. Some free software included with digital cameras does read only specific DNG files from this camera manufacturer (e.g. SilkyPix).
    In order to read DNG from other cameras the user has to register and pay this software.
  2. Some software will not open DNG files from new camera models which are not yet supported.
  3. Some software cannot interpret the sensor data correctly, because the camera model is not yet supported. This can result in very different issues, e.g. brighness or color deviations. I experienced this for K-30 DNG files with CaptureOne v6.4. Only C1 v7.x seems to be able to read these DNG files correctly.
Important is, that you carefully check, that a software is really using the raw data in the DNG and not just displaying (or even processing) the embedded JPG preview. In order to see the difference I am replacing the full resolution preview in the DNG files from my K-30 allways with a medium preview. With this you can see immediately what a software is reallly doing. And you can reduce DNG file size a little bit.

So the fact is, that DNG files can contain proprietary camera data, which needs correct interpretation. This is a specific engineering effort, which requires time and communication with camera manufacturer. To my humble understanding DNG is just a standardized container (like ZIP file), which can contain proprietary raw content and non proprietary meta-text information.

Peter

The entire purpose of DNG was to remove the problem of new formats having to be understood.
DNG is a royalty-free format and no licensing is required to read nor to write the files.
I would be very surprised is any software capable of handling DNG had a problem reading files such as those created in-camera by Pentax cameras.

DNG does have some special capabilities that are not fully supported by a lot of software.
- One of which is much like the ZIP container you mention, and I have NO IDEA why it would ever be used, is where the original RAW file is actually embedded in the DNG.
This results in the DNG raw file existing in a generic format, along with the proprietary camera manufacturer raw format. This will essentially double the size of the file, and as far as I am aware, there is no reasonable benefit for doing so.
- A second DNG format is Adobe's lossy-compressed DNG file. So far a lot of software doesn't know how to handle this.
- The spec allows for a lot of really odd stuff, but I'm not aware of any actual wide spread uses that would cause problems, *unexpectedly*!

Beyond those two examples, I'm not sure if you're having bad luck form experience or are if you're simply getting bad information off the Internet. It sounds like a lot of False information.

I would suggest seeing if you can get more current versions of any software which didn't work in the past. Some of the older software put out 5 to nearly 10 years ago when DNG was first created did not adhere to the specs and was not as flexible as the specs required. I cannot imagine any software written that poorly would still be around without some updates.
11-12-2013, 08:45 AM   #14
Forum Member




Join Date: Aug 2012
Photos: Albums
Posts: 50
Original Poster
QuoteOriginally posted by amoringello Quote
...I would be very surprised is any software capable of handling DNG had a problem reading files such as those created in-camera by Pentax cameras...
CaptureOne v6.4 definitively has big color problems with DNGs from K-30.
I was told that v7.x has no longer this limitation, but this version does not run with Windows 32bit.

See discussion and my image examples dated "08-02-12, 04:44 PM"
https://www.pentaxforums.com/forums/151-pentax-k-30-k-50/194485-dng-format-issues.html
I had a long discussion with PhaseOne about it and they reported, that every DNG content does need manufacturer specific interpretation.
Without information from manufacturer or deep research and investigation the rendering result will be limited.
Due to this some software rejects DNG from unsupported cameras. Other SW renders DNGs, but with weak result.

Peter
11-12-2013, 04:50 PM   #15
Veteran Member




Join Date: Oct 2013
Posts: 402
really educational thread. I always thought that the editing was done on a temporary file copy and that the "Save As" protected the original ,, but now i am learning that is not the case.

Also,, be careful using Picasa to convert raw files ,, it edits EXIF and gives itself ownership of the image.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
dam, dng, e.g, files, format, k-30, k-50, lightroom, pentax k30, pentax k50, tools, xmp
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
Managing unwanted RAW (DNG) files when shooting in RAW+ Weevil Digital Processing, Software, and Printing 3 03-24-2013 07:43 PM
PEF files converted to DNG with Lightroom not readable by Raw Therapee? klh Digital Processing, Software, and Printing 13 08-08-2012 06:13 AM
If I set my camera to RAW - use PEF or DNG? hambone Pentax K-5 21 01-25-2012 08:58 PM
Batch Conversion/Extraction DNG to JPEG? go4java Digital Processing, Software, and Printing 2 04-04-2010 08:32 AM
Help! Batch Conversion - from .dng to .jpg J.Scott Digital Processing, Software, and Printing 9 02-06-2010 04:03 PM



All times are GMT -7. The time now is 09:34 AM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top