[RobG]

The camera can shift the sensor really quick, a few milliseconds and it's done. It should work for any shutter speed.

If I'm calculating correctly, a shutter speed of 1/2000s is an exposure time of 0.5ms. A shutter speed of 1/60s is an exposure time of 16.7ms. For fast shutter speeds you'd need to shift the sensor in the sub microsecond range. Even a relatively slow shutter speed doesn't given you a lot of time for multiple shifts.

That is if the camera has an electronic shutter. Also the user is really not allowed to move the camera at all (unless the SR balances it out). We are talking about very small movements that can ruin the effect, so I think it is safe to assume that this really mostly makes sense for tripod use.

Seems like you would use an electronic shutter and take a series of photos, really similar to the HDR jpeg mode that is currently available. Moving the sensor at tiny bit left or right shouldn't take but a couple of miliseconds at most.

Right, my original assumption was that the camera would capture multiple frames without operating the physical shutter (full sensor captures in movie mode essentially), but the most likely way of doing it would be with multiple exposures using the physical shutter, slowing down the process but making it easier to achieve in firmware. I'd be really surprised if it could be done with the SR still being active (because of the precision of motion required), so it would really require a tripod. Still, it would be fascinating to have a way to effectively upscale the sensor.

[Gimbal]

If I'm calculating correctly, a shutter speed of 1/2000s is an exposure time of 0.5ms. A shutter speed of 1/60s is an exposure time of 16.7ms. For fast shutter speeds you'd need to shift the sensor in the sub microsecond range. Even a relatively slow shutter speed doesn't given you a lot of time for multiple shifts.

I'm not quite following you, the sensor shift is done between the exposures. As long as the scene doesn't change it wouldn't matter if the sensor shift took five minutes.

[Class A]

And there is no way the resolution would increase 4x, not even 2x. I'll give it an 1.5x improvement under ideal conditions, talking from experience.

Have you performed pixel-shifting (after demosaicing) or sensel-shifting (before demosaicing)?

If half-sensel shifting is performed in a manner that each virtual quarter sensel is covered by a quarter of an R, G, and B sensel then there should be no need for demosaicing at all (which improves colour resolution) and there will be some gains for spatial resolution due to the virtual quarter pixels (nowhere near 2x, though).

Not having spent much time thinking about this, I'd definitely try full sensel-shifting (which will just address colour resolution) and then compare the results to half-sensel shifting (which has theoretical advantages, but will require adequate post-processing to yield the respective results).

EDIT: Interesting analysis of the Olympus E-M5 II's high-resolution mode: "...apparently adding 10-12% to MTF50 grayscale linear measurements at f/5.6..."


Note that the 16MP and 64MP line indicators in the legend should be swapped.

[RobG]

I'm not quite following you, the sensor shift is done between the exposures. As long as the scene doesn't change it wouldn't matter if the sensor shift took five minutes.

Please read the rest of what I said:
"Right, my original assumption was that the camera would capture multiple frames without operating the physical shutter (full sensor captures in movie mode essentially)"

It depends on how the sensor shift is implemented. If it's a variation on the way the in-camera HDR works, no problem, but it would increase the requirement for having the camera on a tripod. The time gap between physical shutter frames is much longer and the wear and tear on the shutter mechanism is higher.

[kadajawi]

Apple rather arbitrarily decides when you hardware is too old. I'm using an iPhone three, and I'm sure I'm getting close, but so far, everything runs great.

My guess is Apple will release any updates to Aperture needed for OS upgrades for a while, they just won't spend money on new features.

But. my old Core 2 duo 7 years ole still runs on10.6. It still does way more than I bought it to do.... you may not be able to do whatever new thing they think is great. But, with the last upgrades, I didn't notice a single improvement... though they claim to have made 200 of them.

Yup. Well, sometimes it's really underpowered hardware that causes the lack of updates.


My mother has a 9 year old Mac Mini running 10.6.8, and it runs well. Only recently have I started to run into apps that won't run on it anymore (it's a 32 bit CPU, so I'm stuck with 10.6, and some apps won't run on it). Of course compared to the Windows world that's pathetic, a computer like that should easily run Windows 8.1 or soon-ish Windows 10. But not too bad.


As for phones I think Apple is more consistent in updating their software, as compared to the Android world which is all over the place. Some devices see updates over a long time (though long essentially means 2 years in this world), others won't be updated at all.


Though compared to Pentax' lack of support almost anything is good.

[Racerdew]

Well said Kadajawi,
you explained it better than I did.

[interested_observer]

Superresolution is old tech.

Everybody can play with it, e.g., try
-> PhotoAcute Studio Example :: Processing the images taken using tripod, with trees swinging in snowstorm, with no PhotoAcute Studio profile for the given camera/lens.

However, I wouldn't expect that much resolution increase from the superresolution approach. Not with Bayer filter sensors (monochrome sensors could perform a bit better).

And there is no way the resolution would increase 4x, not even 2x. I'll give it an 1.5x improvement under ideal conditions, talking from experience. Certainly no killer feature.

Evening Falk,

I do agree with your assessment. Photoacute has some excellent data and analysis showing any where from 1.3 to about 1.8x improvement. However, their analysis is only for stacking images, pixel on top of pixel (essentially HDR, noise reduction). The main difference I see here is the application of a sensor shift to fill in the spaces between the pixels, which brings in new and additional information. Photoacute does not address pixel shifting. This should actually add to the base raw resolution - beyond that of just stacking pixels. This is a capability that has not really been available nominal camera users up until now.

In doing a bit of literature searching - a number of papers up through about 2010, was showing a 1.3 to 1.5x improvement in resolution. These papers tended to look at single row sensors using pixel shifting, which appear to impose limitations. I am sure that there are other papers based on sensor movement, I just did not come across them. Then after 2010, papers started to use whole sensor shifting. One paper in particular that took the approach of effectively shifting the entire sensor array (at least that is my interpretation), and evaluated various pixel positioning strategies. This one paper went as far as applying it to an older Canon Rebel (CCD) to confirm their findings of 4x and 8x increase in resolution. The 4x tends to align with what Olympus is claiming.

• http://web.media.mit.edu/~shiboxin/files/Shi_ECCV14.pdf

There are a number of industrial cameras that utilize this approach. One manufacturer, claims their 1280 × 1024-pixel array of a 1/2-in. color sensor. Under software control, 1-, 3-, 5-, 12-, and 21-Mpixel resolution can be achieved, based on the number of sensor movements and pixel positions made. The more movements the greater the resolution. They also claim that resolution can be increased through additional subpixel movements of the sensor until the limits of the resolving power of the optics are reached. Its interesting to note that these are industrial cameras for industrial purposes, not photographic instruments used for fine art photography. I do think that there is a difference.

• Pixel shifting increases microscopy resolution - Vision Systems Design

Then there is this interesting analysis from Martin Doppelbauer. He presents an excellent analysis on the topic. The bottom line from his conclusions is "Pixel-shifted sensors do provide an improvement over unshifted sensors but they don't get anywhere near high-resolution sensors".

• Pixel-Shifting Sensors

Overall, I do think that the pixel shifted results will be an improvement over the native base sensor. However, the next size up in both sensor size and resolution will be safe. That is why, I think that the 645D/Z is safe from a super-resolving smaller sensor. Again, in the Camera Store video comparing the Olympus 4:3 to the 645Z looked to be very good, very close - but there was no real critical analysis presented in terms of image quality.

I think that its a worthwhile effort and the results will show a marked improvement, but I don't think that Pentax is going to loose any 645D/Z sales, because sensor shifting is too good. If it was the nirvana, they would not be offering this capability.

I came across this video overview from another camera manufacturer...

You Tube

You Tube

---

Last edited by interested_observer; 02-09-2015 at 09:59 PM.

[gazonk]

Ahh, but remember what Apple did with their Maps application? Even though the application could run on the older iPhone 4 Apple wanted people to upgrade so they locked the app out from that device. People cited Google Maps as the baseline but that didn't sway Apple.

Please correct me if I'm wrong.

You're wrong. Only the 3D city view wouldn't work on the 4 with its older processor.

[enyceckk]

Really hope they add this to K3. I would be one happy customer =).

Seems like they have this tech back in the year 2000. It's the RDC-7 which can take 7MP super high resolution by shifting the sensor.

http://www.steves-digicams.com/rdc7_pg3.html Maybe there is hope for old SR cameras? lol


The RDC-7 offers 7-megapixel "Pro" modes for capturing high resolution still images. To quote Ricoh:

"There are three Pro Mode options-all based on Ricoh's innovative Image Enhancement Technology. Pro-L mode shifts the CCD by one pixel to take two shots that a Ricoh-developed algorithm composes into one image. This boosts resolution and definition by 20% without increasing image size.



The default Pro Mode uses a Ricoh-developed interpolation algorithm that eliminates "jaggies" when the output resolution increases to 7 megapixels. The resulting image size is 3,072 x 2,304 pixels. Pro-H mode raises output resolution to a maximum 7 megapixels by simultaneously incorporating both the default Pro and Pro-L modes."

The time required for "in camera" processing of these Pro mode images ranges from 30 seconds to a minute and a half. Pro-L and Pro-H modes require the camera to be mounted on a tripod and work only with stationary subjects.


Sample image of the 7MP super resolution (Full size http://www.steves-digicams.com/sdc-classic/rdc7/samples/r0010031.jpg)







There are software that can do super resolution called PhotoAcute.


Samples:

[falconeye]

Have you performed pixel-shifting (after demosaicing) or sensel-shifting (before demosaicing)?

If half-sensel shifting is performed in a manner that each virtual quarter sensel is covered by a quarter of an R, G, and B sensel then there should be no need for demosaicing at all (which improves colour resolution) and there will be some gains for spatial resolution due to the virtual quarter pixels (nowhere near 2x, though).
...
EDIT: Interesting analysis of the Olympus E-M5 II's high-resolution mode: "...apparently adding 10-12% to MTF50 grayscale linear measurements at f/5.6..."

Hi,

thanks for the intersting thought and info. The 12% increase in MTF50 would correspond to 1.4 - 1.5x resolution increase, by accident what I guessed. And it was a pure guess talking from experience.

However, when it comes to resolution I fear an MTF50 analysis is flawed. One rather should study what happens to MTF10. E.g., a D800 and a D800E have much different resolutions (when you look at DxO M-Pix data which are similiar to an MTF50 analysis). Still, the actual resolution limits do not differ at all, the D800 just needs a tad more sharpening applied. Ie., there are too many traps involved when people discuss superresolution. Same applies to tele converters vs. cropping, or Bayer-AA filter vs. no filter btw ...

And I did pixelshifting (after demosaicing). PhotoAcute doesn't have support for anything else. Nevertheless, I assume sensel-shifting (before demosaicing) to yield worse results, due to color fringing artefacts. As an exercise, I recommend to try to overlay the RGGB subimages of a raw file in a way which doesn't produce color fringing. You'll fail and appreciate the cleverness of current demosaicing algorithms. My guess is, to sensel-shift before demosaicing will end up in a complete mess.

Evening Falk,
I do agree with your assessment. Photoacute has some excellent data and analysis showing any where from 1.3 to about 1.8x improvement. However, their analysis is only for stacking images, pixel on top of pixel (essentially HDR, noise reduction). The main difference I see here is the application of a sensor shift to fill in the spaces between the pixels, which brings in new and additional information. Photoacute does not address pixel shifting.

Photoacute superresolution is based on subpixel information, NOT pixel stacking.

The common practice is to use a tripod which isn't too stable and trigger up to 16 exposures by touching the trigger. This will shift the sensor. PhotoAcute then computes the actual shift by alignment analysis and creates a single superresolved image of 4x resolution. The same procedure can be used to reduce noise and create an HDR image. So, it can be easily confused. Moreover, in any numeric analysis it should be noted that overlaying images reduces their effective ISO number (e.g., with 8 exposures, ISO 125 drops to ISO 16). Lower ISO images have less noise and can be sharpened better (the real enemy for deconvolution sharpening is noise because the method isn't numerically stable). Therefore, much of any measured resolution increase would come from noise reduction, not subpixel information (assuming the images are sharpened individually to best result).

There are software that can do super resolution called PhotoAcute.

Yes, I said so above

[Class A]

Hi,

The 12% increase in MTF50 would correspond to 1.4 - 1.5x resolution increase, by accident what I guessed. And it was a pure guess talking from experience.

Thumbs up.

However, when it comes to resolution I fear an MTF50 analysis is flawed. One rather should study what happens to MTF10.

Agreed.

E.g., a D800 and a D800E have much different resolutions (when you look at DxO M-Pix data which are similiar to an MTF50 analysis). Still, the actual resolution limits do not differ at all, the D800 just needs a tad more sharpening applied.

So are you saying that leaving out a Bayer-AA filter just has disadvantages (potential for moiré) and no real advantages?

I'd love a variant of the new FF model to be available with a Bayer AA-filter, but I know that there is practically zero chance for this to happen and that I'll have to make do with the not quite satisfactory replacement called the AA-simulator.

Nevertheless, I assume sensel-shifting (before demosaicing) to yield worse results, due to color fringing artefacts. As an exercise, I recommend to try to overlay the RGGB subimages of a raw file in a way which doesn't produce color fringing. You'll fail and appreciate the cleverness of current demosaicing algorithms.

I believe I have read enough about a number of demosaicing algorithms to appreciate their cleverness, but also to know that they are all wrong in their own specific way. For very specific scenes a specific algorithm can be optimal, but there obviously cannot be a real substitute for capturing full colour information in the first place.

My guess is, to sensel-shift before demosaicing will end up in a complete mess.

I don't understand why you think that way.

Shifting sensels around by full sensel size increments, emulates full RGB capture for each sensel position (for still scenes). This obviates the need for the always incomplete demosaicing process, providing better colour resolution to start with.

Doing the same, i.e., capturing full RGB information for virtual sensels shifted by half a sensel size vertically and horizontally, provides another image that requires no demosaicing but, in combination with the first image capture, provides more spatial information (in a defined way, unlike the random shifts PhotoAcute has to rely on).

[zoolander]

Originally posted by Rondec
Just try to do this with your in lens image stabilization...
It can be done! But it's called "Supersoftness" instead.

You crack me up ! That was a good one !

---------- Post added 02-11-15 at 07:11 PM ----------

so that’s why new FF is looking like 67...

Looks like you're onto something ! I was wondering the same thing when I stumbled onto this thread. It makes sense.

[falconeye]

So are you saying that leaving out a Bayer-AA filter just has disadvantages (potential for moiré) and no real advantages?

No, that's not what I am saying.
Requiring more sharpening is a bad thing in the first place, only possible at low iso and good image quality otherwise.
What I said is leaving out a Bayer-AA doesn't necessarily reveal additional detail.

I don't understand why you think that way.

Shifting sensels around by full sensel size increments, emulates full RGB capture for each sensel position (for still scenes). ...

Doing the same, i.e., capturing full RGB information for virtual sensels shifted by half a sensel size vertically and horizontally, provides another image that requires no demosaicing ...

I agree that shifting by full sensels yields a full RGB capture skipping the need for demosaicing. Something Pentax could do anyway too, btw.

However, skipping by half sensels does not lead to images requiring no demosaicing. They still need demosaicing and can't be demosaiced by the normal method, as it isn't an RGGB pattern anymore (looking at a full subpixel image, I mean).

However, you are correct that one can work with the full RGB images. You would need a full set of 4 half pixel images per color channel, i.e., 12 images altogether (say 16 then). The shift values would be:

R
0, 0
0, 0.5
0.5, 0
0.5, 0.5

G
0, 1
0, 1.5
0.5, 1
0.5, 1.5

G (optional)
1, 0
1, 0.5
1.5, 0
1.5, 0.5

B
1, 1
1, 1.5
1.5, 1
1.5, 1.5

I.e., the sensor would have to shift up to (1.5,1.5)px away to fetch enough subpixel color information.

If you do that, then yes, sensel shift would a superior method. However, I fear most people don't mean that when talking about superresolution by sensor shift.

Even Olympus doesn't, I fear. Otherwise, they would advertize their ability to get rid of demosaicing and its artefacts.

[Doh]

As a K3 owner the FF pixel count has to reach 50Mp on a FF to give me the same approximate image (circle) on the sensor that would be given by my DA lenses. I suspect that the FF pentax will have a 36Mp sensor so bumping up the pixel count by sensor shifting would make sense.

[Class A]

Requiring more sharpening is a bad thing in the first place, only possible at low iso and good image quality otherwise.

Thanks, should have thought of that.

If you do that, then yes, sensel shift would a superior method.

Well, it is the only senselble way, isn't it.

Eliminating demosaicing must always come first, AFAIC, and then one can essentially produce as many slightly shifted versions of RGB images as one wants. The achievable amount of resolution enhancement would only depend on the time one can afford to spend on the exposure and the precision of the SR mechanism (always assuming still scenes).

However, I fear most people don't mean that when talking about superresolution by sensor shift. Even Olympus doesn't, I fear. Otherwise, they would advertize their ability to get rid of demosaicing and its artefacts.

I don't know what Olympus is doing. Perhaps they just don't want to confuse regular customers with terms like "demosaicing" but do the right thing behind the scenes.