[stevebrot]

Are you saying that the same sensor under pixel shift would result in a DNG file containing four 4000 x 2664 arrays of pixels?

Yes, data describing four complete images, one for each shifted exposure.* The actual pixel count is a little higher, but that is the same even with non-PS RAW. What your computer reports as file properties depends on the sophistication of the codec used to plumb the file. The information is usually pulled from the top-level EXIF, which reports the nominal pixel dimensions for the capture.


Steve

* With appropriate tools you can traverse the TIFF/EP structure and "see" each of the embedded RAW "files" in the TIFF filesystem. ExifTool will give a similar notion since it descends the filesystem looking for metadata and will print out a full set for each of the embedded images.

---

Last edited by stevebrot; 09-06-2016 at 05:22 PM.

[awscreo]

mm, in pentax full review on this site, author stated that with RAW 32gb sd card should hold anout 700+ shots, and 400+ with RAW+. I tried using both raw and raw+, and I'm getting the same number. Was that just a mistake in the review, or i'm doing something wrong?

[stevebrot]

mm, in pentax full review on this site, author stated that with RAW 32gb sd card should hold anout 700+ shots, and 400+ with RAW+. I tried using both raw and raw+, and I'm getting the same number. Was that just a mistake in the review, or i'm doing something wrong?

I think the reviewer must have been smoking something, either that or there have been some improvements in lossless compression. My K-3 can only manage 600 PEF (598 DNG) per 32GB card on initial estimate. That being said, the actual number of files per card if filled may be different than the initial estimate, though 700+ from a 36 Mpx camera seems a bit high. FWIW, the official Pentax specification says 101 36Mpx RAW files on an 8 GB card.


Steve

[awscreo]

I think the reviewer must have been smoking something, either that or there have been some improvements in lossless compression. My K-3 can only manage 600 PEF (598 DNG) per 32GB card on initial estimate. That being said, the actual number of files per card if filled may be different than the initial estimate, though 700+ from a 36 Mpx camera seems a bit high. FWIW, the official Pentax specification says 101 36Mpx RAW files on an 8 GB card.


Steve

Ok then, just a mistake in the article. No big deal) Gotta have to order a 128gb card sometime soon, what worked on my 16mp Olympus doesn't give me much on this monster

[rechmbrs]

If you extract the embedded RAW data (can be done in regular dcraw or dcrawps, the results are identical) you get four quite normal-looking TIFFs based on a standard Bayer-interpolation. Running dcrawps in verbose mode gives a few clues as to what Tom is doing, though a more complete analysis might be done from source or by personal communication with the author, PF user @Tomtor.

https://www.pentaxforums.com/forums/172-pentax-k-3/296497-implementing-automa...hot-dcraw.html


Steve

He doesn't explain the option of getting the four files. I'll try to get an explanation but feel if he does the compositing is probably as good as Pentax ooc or DCU motion compensation. But outputting the four files is useful for quality purposes only.
Be back with what I learn.
RONC

[bar_foo]

The in-camera DNG also have checksums. It is required by the DNG spec and if missing, the file will be rejected by processing software (hash calculated at runtime will not agree with hash from file). If Adobe-generated DNG from PEF are smaller, it is only because the user chose other than the full-size JPEG preview.
Steve

Adobe software can't use the checksum in a from-camera DNG to detect corruption, but it can use that from a converted PEF, so there's something different about the checksums it creates. This allows Lightroom (using "Validate DNG Files") to notice bit rot and other kinds of corruption.
In my tests the DNG produced by ACR (or Lightroom, same engine) was 15-20% smaller than the out-of-camera one, which is far too big a difference to be due entirely to the preview. I suspect that this is because a PC has far more processing power than the computer in the camera, and time is less of a constraint, so it can do a more computation-intensive ZIP compression (the DNG spec allows for different compression modes).

At least this is the case for my K-5 II, I'd be curious to hear what results other models get. The basic file size experiment is easy: take a pictures in PEF, then switch to DNG and take the same picture, repeat a few times, then convert all the PEFs to DNG and compare file sizes. To correct for different-sized previews, you could recreate them in Lightroom with the same settings.

[emalvick]

Like in Tiff files, there are options when generating DNG files (with Adobe's DNG converter) to vary the compression level. In fact, the DNG converter even gives the option of lossy-compression (shouldn't be used by anyone). I've not used the converter in ages, so I'm not sure how any of these options are presented. I could imagine that by default they would go for the maximum lossless-compression.

A camera probably won't try that because of time. The higher the compression, the longer it takes to compress the file. The time is probably negligible in general (i.e. on a shot by shot basis), but when you are trying to shoot high-speed continuously, you don't want in-camera compression to stop you from getting a shot or putting extra lag between shots.

One last thing, Pentax does do some compression for all its files (at least since the K5). Generally, photos with less color variation and lower iso are smaller than colors with more detail, color, and noise. As a result, with my K3, I often get 10-30% more photos on a card than what the camera thinks it will get from the beginning (shooting RAW only).

[Oakland Rob]

Adobe software can't use the checksum in a from-camera DNG to detect corruption, but it can use that from a converted PEF, so there's something different about the checksums it creates. This allows Lightroom (using "Validate DNG Files") to notice bit rot and other kinds of corruption.
In my tests the DNG produced by ACR (or Lightroom, same engine) was 15-20% smaller than the out-of-camera one, which is far too big a difference to be due entirely to the preview. I suspect that this is because a PC has far more processing power than the computer in the camera, and time is less of a constraint, so it can do a more computation-intensive ZIP compression (the DNG spec allows for different compression modes).

At least this is the case for my K-5 II, I'd be curious to hear what results other models get. The basic file size experiment is easy: take a pictures in PEF, then switch to DNG and take the same picture, repeat a few times, then convert all the PEFs to DNG and compare file sizes. To correct for different-sized previews, you could recreate them in Lightroom with the same settings.

DNGs produced in camera do not contain checksums. Per Ms Kost: "Note: Only DNG files created by Adobe software can be validated (camera-‐created DNGs cannot be validated because they do not contain the necessary checksum)." DNG Verification in Lightroom 5 « Julieanne Kost's Blog

Using full sized preview and no compression (and not embedding the RAW obviously) an Lr-generated DNG from a PEF is smaller than the PEF and smaller than the camera DNG of the same scene with same camera settings by a tiny bit (34.8 vs 34.9). Sometimes more.

---------- Post added 09-07-16 at 09:06 AM ----------

He doesn't explain the option of getting the four files. I'll try to get an explanation but feel if he does the compositing is probably as good as Pentax ooc or DCU motion compensation. But outputting the four files is useful for quality purposes only.
Be back with what I learn.
RONC

Actually no. Outputting a single frame from the PS can be useful if the original complete PS image has artifacts that are too much of a mess to correct; it essentially gives you a bracketed shot and you can then use one as if it were just one in a series. You don't get the PS benefit, but you save the shot.

[stevebrot]

Adobe software can't use the checksum in a from-camera DNG to detect corruption

I believe you are not clear on how the validation is done. That being said, there is a lot of published confusion on the topic and I don't have time to lay it out at present.


Steve

[stevebrot]

Outputting a single frame from the PS can be useful if the original complete PS image has artifacts that are too much of a mess to correct; it essentially gives you a bracketed shot and you can then use one as if it were just one in a series.

This brings up some interesting questions. There are multiple output options including non-interpolated linear versions. It should be possible to pixel-shift process from that output, though a thorough pain to figure out how.


Steve

[aleonx3]

I honestly don't know the difference between the two... but just for compatibility, I use DNG instead of PEF, unless of course there are some advantage to the PEF files (such as retaining more information). I use Silkypix DS Pro for RAW development, so I guess it does not matter (even if I have a camera with pixel-shift feature such as k-1, k-3II and k-70 in the future).

[bar_foo]

I believe you are not clear on how the validation is done. That being said, there is a lot of published confusion on the topic and I don't have time to lay it out at present.


Steve

Maybe I'm not clear about some aspect of this, but I do know that Lightroom can validate DNGs converted from PEF but it cannot validate those shot in-camera.

[stevebrot]

Maybe I'm not clear about some aspect of this, but I do know that Lightroom can validate DNGs converted from PEF but it cannot validate those shot in-camera.

I may not be clear on this as well in regards to the validation feature. Proponents of DNG describe the internal hash as being intrinsic and inviolable. Anecdotal descriptions make it sound like the hash should always be there and that tools should calculate and compare on file load (i.e. best thing since buttered bread). A review of the specification reveals that it is optional and that details regarding creation, use, and intent are absent.

Clearly there is a disconnect. Also clear is that the feature is essentially meaningless if it requires a pass through Adobe software to initialize the tag value (the case for the data already being corrupt on conversion is ignored).


Steve

[bar_foo]

I may not be clear on this as well in regards to the validation feature. Proponents of DNG describe the internal hash as being intrinsic and inviolable. Anecdotal descriptions make it sound like the hash should always be there and that tools should calculate and compare on file load (i.e. best thing since buttered bread). A review of the specification reveals that it is optional and that details regarding creation, use, and intent are absent.

Clearly there is a disconnect. Also clear is that the feature is essentially meaningless if it requires a pass through Adobe software to initialize the tag value (the case for the data already being corrupt on conversion is ignored).


Steve

It's hardly meaningless just because it has to go through Adobe software (which it doesn't, technically: Adobe has released code for doing so in an SDK). As I understand it, DNG was initially conceived as a format for archiving and manipulating RAW files at a time when all RAW files produced by cameras were in proprietary, manufacturer-specific formats. Only later did some camera makers start adding DNG shooting as an option. So the checksum was never meant to detect invalid files at the initial stage, it was meant to stamp the file with a checksum at the moment of conversion. That way, the software could flag files that had been corrupted later, as storage media degraded or some other software overwrote a portion of the file; a flagged file could then be restored from backup. I've had exactly this happen to me: a few files that were damaged somehow, copied over from backup, good to go.
It may be that a camera maker could implement the checksum if they want (again, the code is available), but calculating a checksum on a multi-megabyte file is a fairly computationally-intense activity that it might be impractical to implement in a camera.

[stevebrot]

It's hardly meaningless just because it has to go through Adobe software (which it doesn't, technically: Adobe has released code for doing so in an SDK). As I understand it, DNG was initially conceived as a format for archiving and manipulating RAW files at a time when all RAW files produced by cameras were in proprietary, manufacturer-specific formats. Only later did some camera makers start adding DNG shooting as an option. So the checksum was never meant to detect invalid files at the initial stage, it was meant to stamp the file with a checksum at the moment of conversion. That way, the software could flag files that had been corrupted later, as storage media degraded or some other software overwrote a portion of the file; a flagged file could then be restored from backup. I've had exactly this happen to me: a few files that were damaged somehow, copied over from backup, good to go.
It may be that a camera maker could implement the checksum if they want (again, the code is available), but calculating a checksum on a multi-megabyte file is a fairly computationally-intense activity that it might be impractical to implement in a camera.

I understand, though I am not too keen about a feature that requires a complete dissection, analysis, and restructure of the capture DNG file to write to a new, Adobe-generated DNG. There is little similarity between an Adobe-generated DNG and a Pentax-generated DNG. I would prefer a process where the checksum is calculated and written into a close clone of the original similar to the edits I routinely do on the EXIF using ExifTool.


Steve