[newmikey]

Can you give an example of "any camera potential in the sensor data of newer camera models" that would benefit from later software versions? I don't think pixel shift is supported even now by Adobe.

Now you're getting to the point! You use DNG and so your world of reasoning revolves around Adobe.

I'm trying not to take offense. There has been less improvement in actual RAW development in newer Lightroom versions than you might think. More than that, if you use third party apps like Topaz Suite or Nik Effects, you can bring a lot out of your RAW images. I post images here and I don't think my limitations are related to using older software.

For pixel shifting images I typically am using RAW Therapee which does a significantly better job of processing than does the newest version of Lightroom.

No offense is intended but the same applies to you although you are definitely aware there is a world of capable software outside the Adobe realm.

The fact is that both of you claim LR has not significantly improved since what, 2017 (which was the year the version mentioned in this thread was issued). So there are two types of DNG users, the ones who use it to avoid upgrading LR (or the subscription) and the ones who are aware there is software out there which has surpassed LR but they stick with it because DNG matches so well with the LR way of working.

See, I'm not so much against DNG as a format, but against how it is being used and what it has caused people to (not) do. There obviously are software options out there, paid or free, which outperform Adobe products in sheer raw processing results. There also are software options out there that better support the capabilities of newer cameras - you yourself mentioned pixelshift but I'm sure there are quite a number of other things such as lens correction profiles, different demosaicing algorithms (such as but not limited to the dual demosaic amaze or RCD/VNG4 mentioned by house) and much, much more.

Nevertheless, people stick to DNG for reasons such as "my old software supports it so I do not need to pay for an upgrade" or "my existing software supports my newer camera without me having to wait for an upgrade". Either reason leads to loss of options, loss of processing updates and IQ which is stuck at the level of 5 years ago.

So, I come back to the way I rephrased the original question: why would you not use the manufacturer's proprietary raw format, in our case PEF? What possible reason could there be if there is mega-powerful raw software out there, what's keeping you from actually using it to bring out the best from your raw files? And if you do use that software, what is left of the imagined benefit of DNG over PEF?

But I agree with our respected mod, this thread is going off topic which is somewhat my doing. Apologies and I'll quite here.

---

Last edited by newmikey; 06-30-2022 at 01:47 PM.

[slartibartfast01]

Now you're getting to the point! You use DNG and so your world of reasoning revolves around Adobe.


No offense is intended but the same applies to you although you are definitely aware there is a world of capable software outside the Adobe realm.

The fact is that both of you claim LR has not significantly improved since what, 2017 (which was the year the version mentioned in this thread was issued). So there are two types of DNG users, the ones who use it to avoid upgrading LR (or the subscription) and the ones who are aware there is software out there which has surpassed LR but they stick with it because DNG matches so well with the LR way of working.

See, I'm not so much against DNG as a format, but against how it is being used and what it has caused people to (not) do. There obviously are software options out there, paid or free, which outperform Adobe products in sheer raw processing results. There also are software options out there that better support the capabilities of newer cameras - you yourself mentioned pixelshift but I'm sure there are quite a number of other things such as lens correction profiles, different demosaicing algorithms (such as but not limited to the dual demosaic amaze or RCD/VNG4 mentioned by house) and much, much more.

Nevertheless, people stick to DNG for reasons such as "my old software supports it so I do not need to pay for an upgrade" or "my existing software supports my newer camera without me having to wait for an upgrade". Either reason leads to loss of options, loss of processing updates and IQ which is stuck at the level of 5 years ago.

So, I come back to the way I rephrased the original question: why would you not use the manufacturer's proprietary raw format, in our case PEF? What possible reason could there be if there is mega-powerful raw software out there, what's keeping you from actually using it to bring out the best from your raw files? And if you do use that software, what is left of the imagined benefit of DNG over PEF?

But I agree with our respected mod, this thread is going off topic which is somewhat my doing. Apologies and I'll quite here.

Well I never said I used DNG but you still didn't mention any benefits of newer software. You suggested improvements had been made since Lightroom 6.14 which is Adobe I believe so Pixel Shift isn't one of the improvements. Are there any other improvements?

[MarkJerling]

As this is an interesting topic, but tangent to the thread this discussion was posted in, I've moved it to it's own thread.
If I've missed any posts in the thread that belong here, then please let me know the post numbers and I'll do a further cleanup.
Here's the thread this discussion originated in: New firmware for K3 III and K-1 introduced season specific shooting modes - Page 8 - PentaxForums.com

[RobA_Oz]

Seeing Mark’s taken the very sensible step of moving this discussion to a new thread, I can add my experience with PEF and DNG files in Adobe’s Camera RAW (in CS5) on my 2009 27” iMac. Camera RAW treats both the same from the user viewpoint, except that the return of control to the user, on closing, is a lot faster with PEF than DNG.

I should add that my image processing doesn’t begin or end with Adobe products (I don’t use LIghtroom, either). I also use ON1 on my more recent MacBook Pro, Topaz Studio on both and DCU on either when separating HDR images for processing in Nik or Photomatix. Thanks for the reminder about RAW Therapee, which could be a more satisfying user experience than the clunky and slow DCU.

I haven’t seen much in the later versions of Adobe’s CS variants to attract me to upgrading, just as with using my old IMac, as nice as shiny, pretty new things are. YKMV.

[MarkJerling]

So, I come back to the way I rephrased the original question: why would you not use the manufacturer's proprietary raw format, in our case PEF? What possible reason could there be if there is mega-powerful raw software out there, what's keeping you from actually using it to bring out the best from your raw files? And if you do use that software, what is left of the imagined benefit of DNG over PEF?

Let's turn it around: I'd be super interested to hear what advantages PEF brings over DNG.
So far as I'm aware, a PEF file and a DNG file, as per the output from a Pentax camera, is absolutely identical for pixel data.
I would imagine that a PEF to DNG conversion, done outside the camera and using software such as DCU5 may or may not result in identical raw data, but I have not tested that.

As I understand it, therefore, DNG or PEF, straight from a Pentax camera are both the "manufacturer's proprietary raw format(s)" with the only difference being that DNG follows the file structure as laid down per the Adobe open source DNG specification. I'd be keen to hear if that understanding is incorrect.

[MarkJerling]

Coincidentally, I see this topic was discussed some years ago too.
See: PEF vs DNG - PentaxForums.com

[Ian Stuart Forsyth]

So there are two types of DNG users, the ones who use it to avoid upgrading LR (or the subscription) and the ones who are aware there is software out there which has surpassed LR but they stick with it because DNG matches so well with the LR way of working.

One of the reason as to why I prefer DNG is that many software that create color profiles are only compatible with DNG, so anytime you are going to color calibrate you first must select DNG or for manufactures that do not have DNG as a raw you will always have to convert your raw.*** to a DNG file.

There is also other useful information that is found in the DNG file format one is the as shot neutral this tell you the multipliers used by the camera for WB this can be useful for calculating UniWB

[newmikey]

Let's turn it around: I'd be super interested to hear what advantages PEF brings over DNG.
So far as I'm aware, a PEF file and a DNG file, as per the output from a Pentax camera, is absolutely identical for pixel data.

Correct. If generated by the camera itself, no difference.

I would imagine that a PEF to DNG conversion, done outside the camera and using software such as DCU5 may or may not result in identical raw data, but I have not tested that.

Such a conversion preserves the actual raw sensor data (except for pixelshift possibly) but will necessarily skip some MakerNotes and other not- or badly- documented tags, hinting and other contents of a proprietary raw file.

As I understand it, therefore, DNG or PEF, straight from a Pentax camera are both the "manufacturer's proprietary raw format(s)" with the only difference being that DNG follows the file structure as laid down per the Adobe open source DNG specification. I'd be keen to hear if that understanding is incorrect.

Absolutely correct in your understanding.

I already listed some of the disadvantages of a DNG file in this thread. When you have a DNG file, it could be very well be a straight-out-of-camera raw, identical/equivalent in every possible way with a PEF (or any other proprietary raw format). However, it could also be one of the following, which are definitely NOT original or equivalent to a camera-generated raw:

2) A manipulated raw file stored by LR (or some other software)
3) The result of a conversion of a proprietary raw file to DNG via any of the many existing converters which may or may not result in the loss of proprietary Maker notes
4) Same as (3) but now in a lossy compression format which may even omit some sensor data besides Maker notes
5) A non-raw image file (post demosaicing, post WB or both) stored inside a DNG container
6) Any other variation I have not yet named above

This is not about fear but about certainty. The certainty that a PEF/NEF/CR2/ORF/ARW/etc file comes straight out of the camera which generated it, with only minimal adjustments allowed to its EXIF (such as copyright info or GPS coordinates)

but I also came across another one I posted in the 7 year old thread you referred to, I tend to call it the "one way street argument" and it still holds today. I have seen many posts of people stating they converted their raw files to DNG and now software X won't read them anymore or they find some feature they were not aware their then-camera supported. As they converted proprietary to DNG and deleted the originals, that is no longer open to corrective action:

To me it is no longer a question of which is better (neither probably) or even which is better supported by software (my software supports both) but the question of reversibility (for lack of a better word). On the off-chance someone will ever find some hinting or specific tags which are better represented by either format, I'd hold you shoot in the format that is most open to conversion into the other.

Simply put: you can always, at any given time in the future, convert a PEF into a DNG by using freely available tools (Adobe or other) but you can never, ever, do the reverse. For that very simple reason I shoot in PEF on any camera that support it. As the k-5 IIs does PEF, I use it. As the GR only does DNG, I use that. Both formats convert to tif/png/jpg just fine.

As an afterthought: any new proprietary raw formats are usually supported very quickly by open-source software such as DCRAW. There always is an interim period in which a new proprietary raw format lacks support. I still shoot in that format but would convert to DNG on the PC in the meantime, getting rid of the proprietary files afterwards.

[3by2]

I use dng because I use Windows and Adobe and a mixed bag of Adobe versions. It has some speed, size and compatibility advantages in that environment. Non dng files do work but they are slower on adobe products like Lr, especially when the editing is extensive. On the same note, editing information is contained in the dng, not a seperate xmp, so the file structure is cleaner.

If I want to use proprietary camera software then generally I can't use dng, as it can't be read. Likewise Windows can generally read dng, proprietary formats need a specific camera file update so it can be read in file explorer, I just had to do that for my olympus.

As I understand it, the dng format is simply structured differently, no information from the original raw is changed. You can in fact embed the original raw but it doubles the file size.

[newmikey]

One of the reason as to why I prefer DNG is that many software that create color profiles are only compatible with DNG, so anytime you are going to color calibrate you first must select DNG or for manufactures that do not have DNG as a raw you will always have to convert your raw.*** to a DNG file.

Most software that creates color profiles will be perfectly able to create both DCP (initially intended for use with DNG) as well as ICC (for use with any other raw format) profiles. There might be software out there which is limited to only DCP profiles but more and more raw converter software will deal with either type of color profile perfectly fine. RawTherapee is a good example (see below). I therefore fail to understand why you feel such profiles are "only compatible with DNG" - that seems like incorrect information.

RawTherapee allows you to use custom DCP (Adobe DNG Camera Profile) or ICC (International Color Consortium) color input profiles. These can be tailor-made to specific scene conditions to provide the most accurate color rendition, or to generally improve upon the standard matrix profiles available.

RawTherapee ships multiple high-quality, custom-made, general-purpose DCP profiles that can be automatically matched when an image from a supported camera is opened. The available profiles are found in the dcpprofiles folder in your installation directory. It is possible to add your own DCP or ICC profiles to this folder. Matching only works on the exact name of the camera (case-sensitive) as is present in the image metadata.

Most provided DCP profiles are dual-illuminant (see below) and some provide tone curves and looks as well. New profiles are added exclusively based on user submission. Please read How to create DCP color profiles for further instructions in case you want to contribute.

There is also other useful information that is found in the DNG file format one is the as shot neutral this tell you the multipliers used by the camera for WB this can be useful for calculating UniWB

That is something I cannot answer. Perhaps someone else can but I generally set WB myself in post.

[Ian Stuart Forsyth]

Most software that creates color profiles will be perfectly able to create both DCP

Of the major software that I use to create colorprofiles only accept DNG





---------- Post added 07-01-2022 at 01:52 AM ----------

There might be software out there which is limited to only DCP profiles but more and more raw converter software will deal with either type of color profile perfectly fine.

Has nothing to do with this, it is about building DCP and ICC profiles and of the 2 well know require DNG files

There might be software out there which is limited to only DCP profiles but more and more raw converter software will deal with either type of color profile perfectly fine. RawTherapee is a good example (see below). I therefore fail to understand why you feel such profiles are "only compatible with DNG" - that seems like incorrect information.

I am not saying that raw converters and the use of DCP ICC files have a problem dealing with other raw formats, It is the software that is creating the DCP ICC files that are then load into your raw converters

---------- Post added 07-01-2022 at 01:54 AM ----------

RawTherapee allows you to use custom DCP (Adobe DNG Camera Profile) or ICC (International Color Consortium) color input profiles. These can be tailor-made to specific scene conditions to provide the most accurate color rendition, or to generally improve upon the standard matrix profiles available.

Just as they say you can use custom DCP or ICC profiles but it is the software that build these color profiles that use DNG files

---------- Post added 07-01-2022 at 02:03 AM ----------

Now there are ways around this to build profiles on is using raw digger to build data points that then can be used in software like free Argyll and DCamProf

but this can be time consuming and really is not geared towards building profiles for pleasing color production or matching that pleasing color from one camera to another

---------- Post added 07-01-2022 at 02:13 AM ----------

That is something I cannot answer. Perhaps someone else can but I generally set WB myself in post.

Uniwb is used so that one can depict or neutralize the multiplies used to create a white balance from the recorded image, this is so that we have a raw histogram as to how the 3 color channels are accurately being recorded.



here no WB is being applied to the file and gives you a representation as to how the image was recorded within the raw file, depending on the model of camera these multipliers are different

Here is what a Uniwb looks like and is used in understanding what color channels are really being clipped, what is being clipped by the software in the raw converter, color space and even the color profile selected in the conversion.

but this can be time consuming and really is not geared towards building profiles for pleasing color production or matching that pleasing color from one camera to another

I use DxO photolab 5 and it enables you to simulate the colour renderings of various camera bodies when processing the dng files.
*Forgive me if this is not what you were referring to*

[Rondec]

DNG and PEF are containers that should hold the information that the camera brand chooses to store in them. Why Pentax would choose to make their PEF and DNG raw images significantly different is beyond me. It seems unnecessarily complicated.

Adobe does have certain levels of processing that are applied when you open RAW images, but if you are concerned by that, then you would do well to make use of a different RAW editor like Raw Therapee or ON-1, but that isn't an argument about containers, it is an argument about the software that processes them.

[newmikey]

Of the major software that I use to create colorprofiles only accept DNG





---------- Post added 07-01-2022 at 01:52 AM ----------

Of course X-rite ColorChecker software deals also with ICC profiles and non-DNG files:
i1Profiler to Create ICC Profiles for Imaging & Prepress | X-Rite Blog
X-Rite Announces ColorChecker Passport Photo 2 and ColorChecker Camera Calibration Software 2.0 for ICC Profiles

---------- Post added 07-01-22 at 12:48 PM ----------

DNG and PEF are containers that should hold the information that the camera brand chooses to store in them. Why Pentax would choose to make their PEF and DNG raw images significantly different is beyond me. It seems unnecessarily complicated.

Absolutely and I never claimed anything else. Pentax DNG equals Pentax PEF in every possible way.

Adobe does have certain levels of processing that are applied when you open RAW images, but if you are concerned by that, then you would do well to make use of a different RAW editor like Raw Therapee or ON-1, but that isn't an argument about containers, it is an argument about the software that processes them.

I do use a different raw editor, so much seems pretty obvious from my profile. I do NOT have an argument about the software that processes the DNG container but about the container itself, primarily because it may- or may not be a raw file straight out of a camera which supports that container format (DNG).

But we digress, The original question which triggered this whole exchange was:

Why do you use PEF rather than DNG? I’m curious.

And there were more than enough reasons given, starting with the counter question "why would I not use PEF?". I have only heard loads of arguments explaining why OOC DNG is not worse than OOC PEF or at least equal to OOC PEF, but still none which would show it is any better either (unless you are locked in to Adobe software of course in which case I understand the buyers bias). Besides all of the question marks which come with a non-OOC DNG file "in the wild" so to speak, it all comes full circle to the fact I can choose to convert all of my PEFs to DNG at any given time, should the necessity (temporarily) arise but I can never, ever, convert DNGs to PEF. The net sum is that a non-Adobe user stands to lose more, long-term, by using DNG instead of PEF.

But as said, I think it is time to let this thread rest. I'll always be open to people who can show me why I should NOT use Pentax's own raw format with which the camera comes preconfigured. Or for that matter, why anyone using any other brand should not use their brand's proprietary raw format. A reason that, as you say is "an argument about containers" rather than "an argument about the software that processes them"

[newmikey]

Here's a little something someone sent to me just now (my emphasis in bold and red). I thought it was worth adding to the end of this thread:

Quality and file size.
DNG files are generally 15-20% smaller than RAW files, but the difference in quality is minimal. A DNG file can also save the original RAW file within itself. This doubles the size, but is also a nice safeguard.

DNG files also strip out select information to decrease the size and simplify storage and editing. This may include JPEG previews, GPS information, metadata, and certain camera models’ focal points, lighting, and picture controls.

The source? Adobe itself - DNG vs. RAW: Which is better and why? | Adobe

Of course this is not the OOC DNG but the DNG as saved by LR