[MadMathMind]

I've just done a couple comparisons of the star ratings on a k5iis, the 4 star rating is on the absurd level of extra MB for miniscule quality gain. On one of my images, the 4 star gave a 9.6MB image, 3 star gave a 6.3MB image. I stacked them in photoshop and had it take the difference of the two versions. 54.3% of the pixels matched, 87.77% were off by 1 level or less, 97.48% were off by 2 levels or less, 99.59% were off by 3 levels or less. Results may vary depending on the content but in practical working terms, there's not much difference for the image I used...

This is my observation as well, based on export after editing in Lightroom. I noticed my original file sizes were almost always 10+ MB but after export with minimal cropping, file sizes dropped to 6-9MB, for the most part. (There are exceptions.)

---------- Post added 02-01-15 at 02:57 PM ----------

They're still using jpeg compression. You can't mess with it outside the jpeg parameters if you hope to be compatible with programs that read jpegs.

No, but the JPEG algorithm has a number of parameters that can be messed that will affect the output quality.

For example, there is a reference way to convert to Y'CrCb but this is not inherently necessary. It is possible to keep the image as 3 separate RGB channels and apply the compression algorithm to each channel. The output will generally be much larger in filesize but not terribly so in quality. This, for example, may be what the 4 star setting of the K-5 series is doing and why it was ultimately removed. (That's a random guess.)

Downsampling is also not mandatory. It is allowable to keep 4:4:4 or go to 4:2:2 or 4:2:0 (most common). The downsampling parameter affects the size of the blocks. For no downsampling (4:4:4), the blocks are 8x8 but in the 4:2:0 sampling they are 16x16. The amount of work at 4:2:0 is therefore half of what it is at 4:4:4, which means that as the megapixels increase, the workload becomes significantly more; it's always double, but double a small amount may be manageable.

The biggest parameter that can be messed with is compression ratio. There's basically infinite variability that can be done there. An older camera may have used 95% for its top setting because the write speed to the SD card was a bit too slow and the cost was a frame in burst mode. Newer cameras with faster components may use 100%; the parameters may also be significantly different from level to level.

[BrianR]

No, but the JPEG algorithm has a number of parameters that can be messed that will affect the output quality..

Of course, which I asked about in my next post. Just pondering on what, if any, differences in the jpeg implementation between the cameras could cause the file size differences (of course sharpening, noise reduction, etc. may also be significant, but I'm just curious about how the jpeg works at the moment.).

I did find some interesting stuff about the Quantization table and how different manufacturers use different choices and how cameras can set it based on the specific image. As you say, this seems to be the most influential parameter.

You've given me more to look into (especially the down sampling), so thanks. The trade offs they have to make to work within the resources of our little magic picture making boxes is interesting.

---------- Post added 02-01-15 at 04:56 PM ----------

Oh man, your question gave me a headache! Ask an electrical engineer, shall we?

Haha. Fortunately the inner workings can be largely ignored by photographers.

[MadMathMind]

Of course, which I asked about in my next post. Just pondering on what, if any, differences in the jpeg implementation between the cameras could cause the file size differences (of course sharpening, noise reduction, etc. may also be significant, but I'm just curious about how the jpeg works at the moment.).

Right. What is done before the compression algorithm makes a big difference. Noise reduction is huge. Random noise does awful things to JPEG efficiency.

To make a long explanation short, JPEG analyzes frequency content and throws out what it deems to be redundant. "Frequency" is generally thought of as an audio term, but any signal can be decomposed into Fourier modes, which are dubbed "frequencies." Anyway, let me explain a bit of how JPEG works in terms of audio--it's more intuitive to understand. Suppose you have a band with two guitars. One is cranked up much louder than the other and suppose one is a bit out of tune. If they play the same note, the frequencies are slightly different. Because the amplitude of one is much lower than the other, JPEG would probably zero out the quieter one. If these notes were truly not important and inaudible to you, then you wouldn't perceive any difference in the compressed versus uncompressed signals.

Noise is a huge problem because it's largely high frequency (HF) content. HF is where the details lie. The compressor can't tell the difference between noise and good details, and if you have the quality cranked up, it wants to preserve what it believes to be details. In fact, eliminating high frequency information is a very quick and naive way to compress something. Because it largely encodes the details, eliminating this content will still make the subject of the picture recognizable.

That's also a lesson in why noise reduction reduces sharpness. Very simple NR does little more than zero out coefficients of high frequency modes. Of course, when that's done, the detail goes with it and...that's why low-quality NR algorithms make your picture look like everything was wrapped in plastic wrap.

I did find some interesting stuff about the Quantization table and how different manufacturers use different choices and how cameras can set it based on the specific image. As you say, this seems to be the most influential parameter.

You've given me more to look into (especially the down sampling), so thanks. The trade offs they have to make to work within the resources of our little magic picture making boxes is interesting.

Downsampling is almost always done at 4:2:0. All digital video uses this (including Blu-ray). It saves a ton of space and given that most of the signal is contained in the Y channel, not generally perceivable to the viewer. If 4:4:4 uses 16 bits, then 4:2:0 would use 12 bits. 4:2:0 would use only 8 bits.

[sterretje]

Oh man, your question gave me a headache! Ask an electrical engineer, shall we?

It's actually mathematics
Something that is definitely not my strongest point

[bavd]

Never expected this many posts on my initial question. Thanks all! So I presume there is no simple answer to this, but i now think that 1) the jpg files of the k3 are bigger on average than those of the k20d.
and 2) they would have been even bigger (as i thought they would be) if the k3 would still have the 4* bestest jpeg setting.

[BrianR]

To make a long explanation short, JPEG analyzes frequency content and throws out what it deems to be redundant. "Frequency" is generally thought of as an audio term, but any signal can be decomposed into Fourier modes, which are dubbed "frequencies."

The math and I are friends from long ago, this is just an application of it that I've always meant to look into a little closer.

Downsampling is almost always done at 4:2:0. All digital video uses this (including Blu-ray). It saves a ton of space and given that most of the signal is contained in the Y channel, not generally perceivable to the viewer. If 4:4:4 uses 16 bits, then 4:2:0 would use 12 bits. 4:2:0 would use only 8 bits.

For what it's worth both the k-5 and the k-3 show 4:2:2 down sampling.

I was curious about how much this actually ends up saving in the final compressed jpeg, so I tried converting a randomish batch of 50 tiffs out of a k5iis using Faststones resizer program, which lets you choose the different down sampling methods.

Compared to 4:4:4 downsampling, the 4:2:2 files shaved off anywhere from 27% to 10%, depending on the 'Quality' setting (ranges from 0-100, these aren't 'binned' as in Lightroom). The highest savings in filesize were at the end quality levels, at 100 it saved 23% and at 0 it saved 27%. The lowest savings were at 'saner' Quality levels, at 90 it saved 10%, at 80 and 70 it saved 13%, which is pretty far from the 1/3 savings in uncompressed data 4:2:2 gives over 4:4:4. It's not really shocking to fall short of 1/3 given what the rest of the jpeg compression is doing.

Not surprisingly, the actually visual difference between the different down sampling methods was pretty hard to spot, so the space savings are probably worth it especially at the cameras highest star ratings (and Pentax Engineers should be better equipped to make this decision than I am)

It's actually mathematics

Mathematics an electrical engineer likely has a good grasp of.

... 2) they would have been even bigger (as i thought they would be) if the k3 would still have the 4* bestest jpeg setting.

I'm no longer convinced of that

[BrianR]

At the risk no one but me cares, I downloaded a bunch more sample supposed-straight-out-of-camera jpegs from Digital Cameras, Digital Camera Reviews - The Imaging Resource!. All the Pentax camera samples I looked at used 4:2:2 down sampling at all star levels (so down sampling doesn't factor into any of the discussion below). They had multiple star levels for the k10d and k20d (the k5iis samples were my own). I compared the Quantization tables and ranked the quality of the compression used.

The short version:

Recent cameras, 645z, k3, k5iis, and k20d all use the same tables at their respective highest star level, which corresponds to the highest possible jpeg quality out of most software. The k10d's highest 3-star quality is the same as the k20d's 2 star, they seriously ramped up the JPEG quality level at this point (at a large cost in filesize). They also jacked up the low end quality ratings at each step k10d -> k20d -> k5iis.


The longer version:

Files out of the cameras are ranked below according to the quality level their Quantization table indicates (nothing numeric, just a relative ordering). Listed on the same row means the Quantization tables were identical. The k5iis and k20d 1-star were pretty close, the rest of the rows indicate a decent change in quality (and a more lossy compression as you move down the rows).

Most shocking to me was the relatively high level of compression out of the 645d. I wanted to believe that the Imaging Resource sample was not at 3 stars, but it's average jpeg file size is supposed to be 17MB, compared to the 645z's average of 30MB, a big enough difference that it looks like they opted for more lossy compression on the 645d. Who buys MF to shoot jpeg? I don't know, but the 645d jpeg was still stunningly impressive in detail.

Given the reorganizing of the star ratings over the years, I would recommend any full time JPEG user with a recent camera compare the low star ratings to the high star ratings to see if the low star versions don't meet their needs just as well, but with a substantial file size savings.

Anyway, my rankings (again, based solely on the Quantization tables, and just relative to one another):

1. K3/645z ***, K20/k5iis ****
2. K20/k5iis ***
3. 645d ***
4. k5iis **
5. k10d ***, k20d **
6. k10d **
7. k5iis *
8. k20d *
9. k10d *

[kh1234567890]

So, jpeg gurus, why is the k-3 file not as large as you'd expect from the increased resolution?

Indeed. JPEGSnoop shows that the compression schemes are very similar.

Obviously having the extra megapixels is pointless since the limiting factor, as far as the amount of detail recorded (and hence compressibility) goes, is the lens. Even a good one