How Pentax Pixel Shifting Works
Super resolution increases image detail while reducing artifacts
By PF Staff in Articles and Tips on Apr 22, 2015
The Pentax K-3 II debuts an innovative feature called the Pixel-Shifting Resolution System, a type of super-resolution technology that increases the effective level of detail in images without requiring a higher sensor resolution.
In this article, we will describe the traditional Bayer sensor implementation and compare it to Pentax's new pixel-shifting system. We will also explain how the technology works and what benefits it can deliver to the photographer.
How Today's Digital Sensors Work
Most digital cameras today— including the Pentax K-3 II and all previous Pentax DSLR models— use color image sensors with pixels laid out in a Bayer pattern. In this layout, each pixel measures the intensity of just one color of light: either red, green, or blue.
Fig. 1: Bayer pattern
This is accomplished by a special filter on top of the sensor that restricts the wavelengths reaching each pixel. The filter layout is uniform and is designed so that there are twice as many green pixels as red or blue pixels, since green light is naturally more abundant.
Fig. 2: The light that reaches each pixel
Thus, on a 24 megapixel sensor like the one in the K-3 and K-3 II, 12 million pixels are sensitive to green light, 6 million are sensitive to red, and 6 million are sensitive to blue. No other color data beyond this is recorded by the sensor at the time of image capture.
Limitations of the Bayer Pattern
But the K-3 II outputs full color images with each of the 24 megapixels containing red, green, and blue color values. How is this accomplished when much of this color data is never actually captured?
Fig. 3: Demosaicing by analyzing neighboring pixels
In a nutshell, the camera restores all the missing color information by making a mathematical guess based on the data found in neighboring pixels. The technical term for this process is demosaicing or interpolation, and it can be carried out by the camera or in RAW development software. Since each pixel is bordered by four pixels sensitive to each of the two missing colors, the camera can usually make a very accurate calculation of the missing color values. Different demosaicing techniques look at a varying number of neighbors in different ways.
Fig. 4: Three different demosaicing algorithms - none can perfectly reproduce the original image
There is unfortunately no optimal demosaicing algorithm, and so Bayer sensors always suffer from a slight loss of detail when reproducing an object, especially near edges, where pixel boundaries play a key role. This is why digital images from Bayer sensors are not perfectly sharp at the pixel level, even when sound photographic technique is applied. Also, when the algorithm fails to correctly guess the missing colors, we can observe artifacts such as moire or zipper patterns. These artifacts can be suppressed by an anti-aliasing (AA) filter, but this comes at the cost of resolution. The AA filter essentially spills light across neighboring pixels (applies a blur) so that their color values are not independent and thus more likely to be interpolated correctly.
In Figure 4, the difference between the original image and that interpolated by Algorithm A (bilinear) is comparable to what a digital camera might actually output.
Fun fact: if you compare an out-of-camera JPEG to a RAW file developed in a third-party post-processing application, there will often be a difference in the way the two have been interpolated, and one might look better than the other in areas containing fine details. Desktop processing is often better since more computing power is available.
Insight into the Pixel Shift Resolution System
Pentax's latest sensor technology seeks to address the limitations of the Bayer pattern. It increases the amount of color data recorded— and thus the sensor's practical resolution— by microscopically shifting pixels around at the time of image capture (up, down, left, and right by one pixel relative to the default sensor position) so that each pixel gets fully exposed to all three color channels across a series of four images. This effectively quadruples the number of red/blue pixels and doubles the number of green pixels, increasing the overall level of recoverable detail in the final processed image. In theory, it should also make the AA filter/AA filter simulator obsolete, since the recorded image data would be similar to that observed by a Foveon sensor (only used in a handful of cameras due to inherent light loss) and thus not as susceptible to moire.
Fig. 5: Pentax pixel shifting in action
Fig. 6: The four-shot process
The net result is that all 24 million pixels on the K-3 II's sensor will capture complete red, green, and blue (RBG) color data without the need for demosaicing. Plus, because more light is captured across the series of images, noise levels could be lower.
Fig. 7: Nominal pixel-shifting results
The output of a 24-megapixel Bayer sensor with pixel shifting could theoretically be compared to the detail reproduced by a conventional Bayer sensor with a resolution between 48 and 96 megapixels if we consider the total recorded color data shown in Figure 7. In fact, Olympus has already used a similar technology to implement super-resolution and increase the effective pixel count, albeit via a slightly different 8-image process. The net result of both approaches should be comparable. We tend to favor Pentax's approach for reasons explained in the conclusion of the article.
Fig. 8: Studio test of super resolution
The implementation of the pixel-shifting system in the K-3 II seems to require the use of a tripod and involves the capture of four back-to-back images which are then synthesized in the camera or by post-processing software. Additional time will surely be required compared to conventional shooting, but we do not yet know exactly how the K-3 II will behave in its super-resolution mode (i.e. whether or not the shutter will stay open during the whole sequence). We will report on this as soon as we have the camera in our hands.
Fig. 9: ISO 12,800 noise comparison
At this point we can only make an educated prediction about the K-3 II's image quality when pixel-shifting is active. We believe that the practical improvement in resolution delivered by the pixel shifting system will be analogous to comparing two cameras with and without an AA filter. Thus, we think the K-3 is to the K-3 II as the K-5 is to the K-5 IIs. As an added bonus, however, the K-3 II will be mostly immune to moire and zipper artifacts without having to rely on the AA filter simulator. Overall noise levels should also be reduced since the multiple exposures offer a larger amount of luminance data for each color channel. Thus, the benefits of pixel shifting should be more significant than the mere removal of an AA filter. This assumes, of course, that the camera is kept perfectly still at the time of image capture. Any motion could backfire and undo or reduce the benefits of this system.
Until pixel-shifting systems can be updated to be less sensitive to camera motion and not require multiple exposures, a sensor with a higher physical pixel count will continue to be more versatile in practice. Even when on a tripod, slight vibrations may prevent the capture of what theory would dictate when applying super-resolution. It will be interesting to see how well the pixel shifting mechanism might play along with Pentax's Shake Reduction system in the future.
If the pixel shifting scheme were revised to only require a single exposure (possibly lasting slightly longer than the selected shutter speed), under the hood, all pixels on the sensor could be exposed to the missing color channels for a fraction of the total exposure time. This could have the potential to deliver results superior to an interpolated image and it could work autonomously behind the scenes.
The Bottom Line
TL;DR: Ultimately, we expect the K-3 II to be able to deliver noticeably more detailed landscape or cityscape images than the K-3 with a lower susceptibility to moire, noise, and edge artifacts. If this system could evolve to work off a tripod in the future, it would be a very significant technological advancement for Pentax!
We believe that making improvements on the pixel level rather than indefinitely increasing image pixel count is the future of digital sensor development, since this will help keep file sizes more manageable and reduce the strain on in-camera processors. While we cannot quantify the K-3 II's practical resolution at this point, we do expect it to deliver a visible improvement at the pixel level that will ultimately allow for larger prints. Since the highest-resolution APS-C sensor currently on the market only has 28 megapixels, the K-3 II may well be able to out-resolve it and thus become the sharpest APS-C camera in the world when used on a tripod.
Figure 1/2 image credits: Wikipedia. Licensed under CC-BY-SA 3.0 | Figure 5/6/7/8/9 image credits: Ricoh Imaging