Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
09-06-2016, 05:16 PM   #16
Site Supporter
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 26,617
QuoteOriginally posted by reh321 Quote
Are you saying that the same sensor under pixel shift would result in a DNG file containing four 4000 x 2664 arrays of pixels?
Yes, data describing four complete images, one for each shifted exposure.* The actual pixel count is a little higher, but that is the same even with non-PS RAW. What your computer reports as file properties depends on the sophistication of the codec used to plumb the file. The information is usually pulled from the top-level EXIF, which reports the nominal pixel dimensions for the capture.


Steve

* With appropriate tools you can traverse the TIFF/EP structure and "see" each of the embedded RAW "files" in the TIFF filesystem. ExifTool will give a similar notion since it descends the filesystem looking for metadata and will print out a full set for each of the embedded images.


Last edited by stevebrot; 09-06-2016 at 05:22 PM.
09-06-2016, 05:32 PM   #17
Pentaxian




Join Date: May 2016
Photos: Albums
Posts: 695
Original Poster
mm, in pentax full review on this site, author stated that with RAW 32gb sd card should hold anout 700+ shots, and 400+ with RAW+. I tried using both raw and raw+, and I'm getting the same number. Was that just a mistake in the review, or i'm doing something wrong?
09-06-2016, 05:56 PM   #18
Site Supporter
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 26,617
QuoteOriginally posted by awscreo Quote
mm, in pentax full review on this site, author stated that with RAW 32gb sd card should hold anout 700+ shots, and 400+ with RAW+. I tried using both raw and raw+, and I'm getting the same number. Was that just a mistake in the review, or i'm doing something wrong?
I think the reviewer must have been smoking something, either that or there have been some improvements in lossless compression. My K-3 can only manage 600 PEF (598 DNG) per 32GB card on initial estimate. That being said, the actual number of files per card if filled may be different than the initial estimate, though 700+ from a 36 Mpx camera seems a bit high. FWIW, the official Pentax specification says 101 36Mpx RAW files on an 8 GB card.


Steve
09-06-2016, 06:06 PM   #19
Pentaxian




Join Date: May 2016
Photos: Albums
Posts: 695
Original Poster
QuoteOriginally posted by stevebrot Quote
I think the reviewer must have been smoking something, either that or there have been some improvements in lossless compression. My K-3 can only manage 600 PEF (598 DNG) per 32GB card on initial estimate. That being said, the actual number of files per card if filled may be different than the initial estimate, though 700+ from a 36 Mpx camera seems a bit high. FWIW, the official Pentax specification says 101 36Mpx RAW files on an 8 GB card.


Steve
Ok then, just a mistake in the article. No big deal) Gotta have to order a 128gb card sometime soon, what worked on my 16mp Olympus doesn't give me much on this monster

09-06-2016, 06:36 PM   #20
Site Supporter




Join Date: Jan 2007
Location: Conroe, TX USA
Posts: 207
QuoteOriginally posted by stevebrot Quote
If you extract the embedded RAW data (can be done in regular dcraw or dcrawps, the results are identical) you get four quite normal-looking TIFFs based on a standard Bayer-interpolation. Running dcrawps in verbose mode gives a few clues as to what Tom is doing, though a more complete analysis might be done from source or by personal communication with the author, PF user @Tomtor.

https://www.pentaxforums.com/forums/172-pentax-k-3/296497-implementing-automa...hot-dcraw.html


Steve
He doesn't explain the option of getting the four files. I'll try to get an explanation but feel if he does the compositing is probably as good as Pentax ooc or DCU motion compensation. But outputting the four files is useful for quality purposes only.
Be back with what I learn.
RONC
09-07-2016, 07:52 AM   #21
Forum Member




Join Date: Feb 2014
Location: Vancouver, BC
Posts: 95
QuoteOriginally posted by stevebrot Quote
The in-camera DNG also have checksums. It is required by the DNG spec and if missing, the file will be rejected by processing software (hash calculated at runtime will not agree with hash from file). If Adobe-generated DNG from PEF are smaller, it is only because the user chose other than the full-size JPEG preview.
Steve
Adobe software can't use the checksum in a from-camera DNG to detect corruption, but it can use that from a converted PEF, so there's something different about the checksums it creates. This allows Lightroom (using "Validate DNG Files") to notice bit rot and other kinds of corruption.
In my tests the DNG produced by ACR (or Lightroom, same engine) was 15-20% smaller than the out-of-camera one, which is far too big a difference to be due entirely to the preview. I suspect that this is because a PC has far more processing power than the computer in the camera, and time is less of a constraint, so it can do a more computation-intensive ZIP compression (the DNG spec allows for different compression modes).

At least this is the case for my K-5 II, I'd be curious to hear what results other models get. The basic file size experiment is easy: take a pictures in PEF, then switch to DNG and take the same picture, repeat a few times, then convert all the PEFs to DNG and compare file sizes. To correct for different-sized previews, you could recreate them in Lightroom with the same settings.
09-07-2016, 08:43 AM   #22
Pentaxian
emalvick's Avatar

Join Date: Feb 2008
Location: Davis, CA
Photos: Gallery
Posts: 1,472
Like in Tiff files, there are options when generating DNG files (with Adobe's DNG converter) to vary the compression level. In fact, the DNG converter even gives the option of lossy-compression (shouldn't be used by anyone). I've not used the converter in ages, so I'm not sure how any of these options are presented. I could imagine that by default they would go for the maximum lossless-compression.

A camera probably won't try that because of time. The higher the compression, the longer it takes to compress the file. The time is probably negligible in general (i.e. on a shot by shot basis), but when you are trying to shoot high-speed continuously, you don't want in-camera compression to stop you from getting a shot or putting extra lag between shots.

One last thing, Pentax does do some compression for all its files (at least since the K5). Generally, photos with less color variation and lower iso are smaller than colors with more detail, color, and noise. As a result, with my K3, I often get 10-30% more photos on a card than what the camera thinks it will get from the beginning (shooting RAW only).
09-07-2016, 09:00 AM   #23
Loyal Site Supporter




Join Date: Apr 2015
Photos: Albums
Posts: 498
QuoteOriginally posted by bar_foo Quote
Adobe software can't use the checksum in a from-camera DNG to detect corruption, but it can use that from a converted PEF, so there's something different about the checksums it creates. This allows Lightroom (using "Validate DNG Files") to notice bit rot and other kinds of corruption.
In my tests the DNG produced by ACR (or Lightroom, same engine) was 15-20% smaller than the out-of-camera one, which is far too big a difference to be due entirely to the preview. I suspect that this is because a PC has far more processing power than the computer in the camera, and time is less of a constraint, so it can do a more computation-intensive ZIP compression (the DNG spec allows for different compression modes).

At least this is the case for my K-5 II, I'd be curious to hear what results other models get. The basic file size experiment is easy: take a pictures in PEF, then switch to DNG and take the same picture, repeat a few times, then convert all the PEFs to DNG and compare file sizes. To correct for different-sized previews, you could recreate them in Lightroom with the same settings.
DNGs produced in camera do not contain checksums. Per Ms Kost: "Note: Only DNG files created by Adobe software can be validated (camera-‐created DNGs cannot be validated because they do not contain the necessary checksum)." DNG Verification in Lightroom 5 Julieanne Kost's Blog

Using full sized preview and no compression (and not embedding the RAW obviously) an Lr-generated DNG from a PEF is smaller than the PEF and smaller than the camera DNG of the same scene with same camera settings by a tiny bit (34.8 vs 34.9). Sometimes more.

---------- Post added 09-07-16 at 09:06 AM ----------

QuoteOriginally posted by rechmbrs Quote
He doesn't explain the option of getting the four files. I'll try to get an explanation but feel if he does the compositing is probably as good as Pentax ooc or DCU motion compensation. But outputting the four files is useful for quality purposes only.
Be back with what I learn.
RONC
Actually no. Outputting a single frame from the PS can be useful if the original complete PS image has artifacts that are too much of a mess to correct; it essentially gives you a bracketed shot and you can then use one as if it were just one in a series. You don't get the PS benefit, but you save the shot.

09-07-2016, 09:48 AM   #24
Site Supporter
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 26,617
QuoteOriginally posted by bar_foo Quote
Adobe software can't use the checksum in a from-camera DNG to detect corruption
I believe you are not clear on how the validation is done. That being said, there is a lot of published confusion on the topic and I don't have time to lay it out at present.


Steve
09-07-2016, 09:53 AM   #25
Site Supporter
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 26,617
QuoteOriginally posted by Oakland Rob Quote
Outputting a single frame from the PS can be useful if the original complete PS image has artifacts that are too much of a mess to correct; it essentially gives you a bracketed shot and you can then use one as if it were just one in a series.
This brings up some interesting questions. There are multiple output options including non-interpolated linear versions. It should be possible to pixel-shift process from that output, though a thorough pain to figure out how.


Steve
09-07-2016, 10:19 AM   #26
Pentaxian
aleonx3's Avatar

Join Date: Jul 2007
Location: Brampton, Ontario
Photos: Gallery | Albums
Posts: 3,880
I honestly don't know the difference between the two... but just for compatibility, I use DNG instead of PEF, unless of course there are some advantage to the PEF files (such as retaining more information). I use Silkypix DS Pro for RAW development, so I guess it does not matter (even if I have a camera with pixel-shift feature such as k-1, k-3II and k-70 in the future).
09-07-2016, 02:13 PM   #27
Forum Member




Join Date: Feb 2014
Location: Vancouver, BC
Posts: 95
QuoteOriginally posted by stevebrot Quote
I believe you are not clear on how the validation is done. That being said, there is a lot of published confusion on the topic and I don't have time to lay it out at present.


Steve
Maybe I'm not clear about some aspect of this, but I do know that Lightroom can validate DNGs converted from PEF but it cannot validate those shot in-camera.
09-07-2016, 05:19 PM   #28
Site Supporter
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 26,617
QuoteOriginally posted by bar_foo Quote
Maybe I'm not clear about some aspect of this, but I do know that Lightroom can validate DNGs converted from PEF but it cannot validate those shot in-camera.
I may not be clear on this as well in regards to the validation feature. Proponents of DNG describe the internal hash as being intrinsic and inviolable. Anecdotal descriptions make it sound like the hash should always be there and that tools should calculate and compare on file load (i.e. best thing since buttered bread). A review of the specification reveals that it is optional and that details regarding creation, use, and intent are absent.

Clearly there is a disconnect. Also clear is that the feature is essentially meaningless if it requires a pass through Adobe software to initialize the tag value (the case for the data already being corrupt on conversion is ignored).


Steve
09-07-2016, 07:48 PM   #29
Forum Member




Join Date: Feb 2014
Location: Vancouver, BC
Posts: 95
QuoteOriginally posted by stevebrot Quote
I may not be clear on this as well in regards to the validation feature. Proponents of DNG describe the internal hash as being intrinsic and inviolable. Anecdotal descriptions make it sound like the hash should always be there and that tools should calculate and compare on file load (i.e. best thing since buttered bread). A review of the specification reveals that it is optional and that details regarding creation, use, and intent are absent.

Clearly there is a disconnect. Also clear is that the feature is essentially meaningless if it requires a pass through Adobe software to initialize the tag value (the case for the data already being corrupt on conversion is ignored).


Steve
It's hardly meaningless just because it has to go through Adobe software (which it doesn't, technically: Adobe has released code for doing so in an SDK). As I understand it, DNG was initially conceived as a format for archiving and manipulating RAW files at a time when all RAW files produced by cameras were in proprietary, manufacturer-specific formats. Only later did some camera makers start adding DNG shooting as an option. So the checksum was never meant to detect invalid files at the initial stage, it was meant to stamp the file with a checksum at the moment of conversion. That way, the software could flag files that had been corrupted later, as storage media degraded or some other software overwrote a portion of the file; a flagged file could then be restored from backup. I've had exactly this happen to me: a few files that were damaged somehow, copied over from backup, good to go.
It may be that a camera maker could implement the checksum if they want (again, the code is available), but calculating a checksum on a multi-megabyte file is a fairly computationally-intense activity that it might be impractical to implement in a camera.
09-07-2016, 10:03 PM   #30
Site Supporter
stevebrot's Avatar

Join Date: Mar 2007
Location: Vancouver (USA)
Photos: Gallery | Albums
Posts: 26,617
QuoteOriginally posted by bar_foo Quote
It's hardly meaningless just because it has to go through Adobe software (which it doesn't, technically: Adobe has released code for doing so in an SDK). As I understand it, DNG was initially conceived as a format for archiving and manipulating RAW files at a time when all RAW files produced by cameras were in proprietary, manufacturer-specific formats. Only later did some camera makers start adding DNG shooting as an option. So the checksum was never meant to detect invalid files at the initial stage, it was meant to stamp the file with a checksum at the moment of conversion. That way, the software could flag files that had been corrupted later, as storage media degraded or some other software overwrote a portion of the file; a flagged file could then be restored from backup. I've had exactly this happen to me: a few files that were damaged somehow, copied over from backup, good to go.
It may be that a camera maker could implement the checksum if they want (again, the code is available), but calculating a checksum on a multi-megabyte file is a fairly computationally-intense activity that it might be impractical to implement in a camera.
I understand, though I am not too keen about a feature that requires a complete dissection, analysis, and restructure of the capture DNG file to write to a new, Adobe-generated DNG. There is little similarity between an Adobe-generated DNG and a Pentax-generated DNG. I would prefer a process where the checksum is calculated and written into a close clone of the original similar to the edits I routinely do on the EXIF using ExifTool.


Steve

Last edited by stevebrot; 09-07-2016 at 10:12 PM.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
adobe software, camera, compression, dng, dslr, files, image, images, lightroom, motion, pef, photography, pixel, post, ps, shot, size
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
PEF or DNG? rlatjsrud Troubleshooting and Beginner Help 10 03-19-2016 06:01 PM
RAW-- PEF vs DNG... difference? CEWren Pentax K-5 9 07-23-2011 10:34 PM
Difference between DNG and PEF? ismaelg Pentax DSLR Discussion 15 06-06-2011 01:16 PM
Pentax/Honeywell...Is there any difference? pickles Pentax SLR Lens Discussion 8 05-29-2010 09:12 AM
Difference between PEF and DNG formats john_mantz Pentax DSLR Discussion 6 09-25-2007 11:24 AM



All times are GMT -7. The time now is 07:09 PM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top