[Torphoto]

BTW, excuse my uneducated question: is there a way to do the pixel binning (effectively, supersampling) on the CMOS level, without having to read out all the pixels?

IT would have to be embedded in the sensor it self, IE hardwired in, but it might be doable at a software level to some extent but how well it would work in that case is known or how much processing power will be needed.


As it looks ( aka from Falconeye ) is that the 14.6 mp sensor is the culprit in not getting higher than 720p at superb quality. We won't know until we see video samples from a 1.0 firmware production camera for sure.

[Class A]

Shall we propose to Pentax in firmware 1.0 to hide enabling of the 1536x1024 mode in a customs setting just like ISO 6400?

After having seen the respective vimeo sequence and with the assumption that the final firmware won't produce a better result, I fully second that motion.

I feel allowing AF during video is the same story. If it isn't trouble free (e.g., if you see AF hunting during a movie sequence), then reviewers will jump at it. If it is not available at all or locked away in "at your own peril" menus then reviewers cannot tear it apart as easily. If it is easy to point out flaws then these can be used to mark down scores. If flaws are absent they cannot be used in such a way, even if that is achieved by removing a feature that may be slightly flawed altogether. I'm not too concerned about criticism regarding missing functionality. It will be hard to criticise the K-7 for not having enough on offer.

I'm not saying that reviewers are evil-spirited, on the contrary I've read many glowing reviews for Pentax products, but we all know sad stories about issues being blown out of proportion and hence, completely unnecessarily, causing ill-effect on potential buyers who don't spend the time required to critically evaluate reviews.

[melander]

I think pentax should scrap 1500x1000 video mode, who will use it anyway, 16/9 720p is the way to shoot on this.
You will shoot it and watch it on your 16/9 tv set. Shoot in 1500 and you get grey edges.

[jogiba]

I think pentax should scrap 1500x1000 video mode, who will use it anyway, 16/9 720p is the way to shoot on this.
You will shoot it and watch it on your 16/9 tv set. Shoot in 1500 and you get grey edges.

1536x1024 scales up full screen on my 24" Samsung monitor in 1920x1200 mode.

[falconeye]

A possibly more subtle approach to address the fringing would be to create separate masks targeting magenta and green color artifacts. This can be done by finding the horizontal gradient with a Sobel operator.

I'm working on the assumption that the color artifacts occur mostly along the horizontal axis, and that magenta and green correspond to opposite gradient directions.

Here's an example using the fire engine capture. Note: the aliasing is left as is.
<img>

paene, this looks like an interesting piece of work. I will have to come back to you about this.

Which software did you use to apply your method?

[jogiba]

I just tried Microsoft's new search engine Bing for K-7 videos and it works much better than Google/youtube . Just move the curser over the video thumbnail and it plays the video then just click to run it in normal view.
Pentax K-7 - Bing Video

[Class A]

I think pentax should scrap 1500x1000 video mode, who will use it anyway, ...

I think it should be retained as a kind of RAW video which needs processing to be useful but offers higher resolution to boot with (e.g., to scale down or treat otherwise) than the other formats.

This assumes that the final firmware will actually produce the colour moire artefacts, maybe Pentax will implement something to smooth out the artefacts. However, I'm not sure that would be optimal for us.

If they chose a kind of blurring then this would preclude any other kind of treatment that is more successful in retaining detail.

I can't seem to find a description of your treatment in terms of reducing the color artifacts.

Falconeye, is there anywhere something that adds to this description of your image treatment approach?

A possibly more subtle approach to address the fringing would be to create separate masks targeting magenta and green color artifacts.

Your treated image looks very good indeed. I did not pixel peep in depth but I did a rough comparison to the original and it seems you manage to suppress the fringing without losing any detail. That leaves anti-aliasing to be done in another step but I think that's great because you sometimes won't need that pass and could retain maximum resolution.

---

Last edited by Class A; 06-04-2009 at 07:04 PM.

[paene]

paene, this looks like an interesting piece of work. I will have to come back to you about this.

Which software did you use to apply your method?

I just did some quick experiments in the GIMP using its Sobel, Convolve, and HSL plugins. More complex stuff would definitely be easier in something like Matlab/Octave, SciPy, etc. I'm far from an image processing expert, so I don't have much knowledge to draw from.

Right now, I'm extracting the horizontal gradient twice in HSL space (once for lightness and once for hue). The negative hue and saturation gradient values are combined to create the magenta mask, while the positive hue and saturation gradient values are combined to create the green mask (i think anyway -- it's too easy for me to confuse the signs )

The general idea seems ok, but as always, the devil's in the details. Threshold values need to be tuned after applying the Sobel operator to obtain an accurate and precise starting point for the masks, which then need to be smeared in the correct direction. Also, strong fringing can probably interfere with the gradient extraction.

Then there's the issue of how to modify the source image. I've just done a simple desaturation on certain hue ranges, but it'd be a good idea to consider local saturation and hue.

I'm sure there must be papers on this topic -- it'd be interesting to see what's out there

[paene]

That leaves anti-aliasing to be done in another step but I think that's great because you sometimes won't need that pass and could retain maximum resolution.

The aliasing is kind of interesting -- as was noted previously, it doesn't seem to be a simple up-sampling from a single 768x1024 source image. Maybe some additional information present? Might be able to do some selective blurring there.

BTW, being interested in the technology and the process, I also hope Pentax keeps the 1536x1024 option because otherwise we'd lose out on a lot of available resolution. But yeah it's one of those engineering vs. marketing decisions, and it might be better for Pentax to leave it out if they can't market around the deficiencies. I like your idea of labelling it as a "raw" mode, though who knows how the market would interpret it

[falconeye]

Falconeye, is there anywhere something that adds to this description of your image treatment approach?

At that time, I used some filter or plugin (don't remember) which desatures purple/green along contrast borders. I cannot remember the exact procedure and never was satisfied, anyway. I lost interest in the K20D burst mode soon after as most others did as well

paene's method is a lot more accurate, actually.

GIMP Sobel, Convolve, HSL ...
[...]
I'm sure there must be papers on this topic -- it'd be interesting to see what's out there

This approach seems to make sense.

As for the papers ... 6 billion people don't seem to be enough

Meanwhile, I have started to write a little program which attempts to correct the artifacts. Doing it in a program gives me all the options...

Maybe, we can cooperate?

Currently, I try to restore the missing info by taking neighboring colors into account, somewhat like what is done in the demosaicing.

But combining this with the Sobel idea may be a good thing to try.

But I found that a method which works uniformly accross a range of content is hard to find, too. And fringing at spots which are burned out probably needs extra treatment as well...

The aliasing is kind of interesting -- as was noted previously, it doesn't seem to be a simple up-sampling from a single 768x1024 source image.

Currently, I extract two "half" 768x1024 images from a frame: one with even and one with odd columns only. The odd and even images aren't identical (so, there is some extra info, probably constructed by the firmware), but they are similiar. The correlation between pixels in the odd and even image is much larger than correlation to left or right neighbor.

Also, I do all my defringing in the half images as fringing there is 1 pixel wide, only

So, what I currently do is amplifying the difference between odd and even image and then smooth-merge them together. There are parameters where I can clearly see less aliasing without loosing any resolution.

One of the problems too seems to be that the sub-sampling matrix seems to pick its values from varying positions within the matrix. So, text can look awkward. In order to overcome this, one would have to know the exact sub-sampling matrix layout and restore an even larger size image first (super-resolution techniques).

There are too many parameters to tune, too. So, below, this is just some intermediate result of my attempt to improve the video. I am not satisfied, yet.

Enjoy

Enhanced 1536x1024 street video (51 MB)

[paene]

Maybe, we can cooperate?

Sure, I'd be interested in helping out. I don't have much free time to spare but it's an interesting problem and I'd be happy to contribute whenever I can.

Currently, I try to restore the missing info by taking neighboring colors into account, somewhat like what is done in the demosaicing.

But combining this with the Sobel idea may be a good thing to try.

There are more optimal edge detection algorithms, but maybe Sobel is good enough for the given task. At this point I'm skirting the limits of what I've learnt -- the stuff I remember anyway! I wish I still had my image processing textbook from way back in undergrad, but I've lent it to a friend (likely permanently ).

But I found that a method which works uniformly accross a range of content is hard to find, too. And fringing at spots which are burned out probably needs extra treatment as well...

Luckily the green/magenta color artifacts can be deduced from what we assume of the sub-sampling matrix. I haven't really looked at the demosaicing in detail, and I'd probably learn a lot trying to derive conditions that cause artifacts from that. A varied set of real-world samples would definitely help.

So, what I currently do is amplifying the difference between odd and even image and then smooth-merge them together. There are parameters where I can clearly see less aliasing without loosing any resolution.

One of the problems too seems to be that the sub-sampling matrix seems to pick its values from varying positions within the matrix. So, text can look awkward. In order to overcome this, one would have to know the exact sub-sampling matrix layout and restore an even larger size image first (super-resolution techniques).

Cool, definitely less color fringing and aliasing! I'm not sure exactly what a "smooth-merge" involves, but would it help to up-sample the source images first, process, and downsample to the final result?

[falconeye]

Sure, I'd be interested in helping out. I don't have much free time to spare but it's an interesting problem

Same boat here

Cool, definitely less color fringing and aliasing! I'm not sure exactly what a "smooth-merge" involves, but would it help to up-sample the source images first, process, and downsample to the final result?

Did you see my trial video?

Do you think this is good enough to invest more time? As I don't know how much better it would get (maybe a lot, maybe not much at all).

#smooth-merge: I have to interlace odd and even half images. In doing so, I blend a bit of neighbors and from the other half image. After I already boosted the difference of odd and even half images.

With a bit more time, I think I will bicubic upscale both half images and shift one half image by a defined distance (where the default would be 1), and then add them.

But a better way would be to directly extract an odd and even images which are both 1525x1024. But I cannot without further knowledge of the sab-sampling matrix.

It is obvious that (with 6x3) there must be a "type A" and "type B" of a sub-sampling matrix (because the Bayer pattern cell size doesn't fit a single sub-sampling matrix). So, we better know how Pentax has filled in the holes in the information they got from the sensor.

BTW, defringing is still probably best done in 768x1024 half images as there, fringes are 1px wide only.

I am now also thinking how one could measeure the position-dependent PSF in video mode (if Pentax doesn't release this info).

Any further ideas?

[paene]

Did you see my trial video?

Do you think this is good enough to invest more time? As I don't know how much better it would get (maybe a lot, maybe not much at all).

TBH I'm probably not the one to ask about that . It does improve the aliasing pretty well in a lot of places. I see what you mean with the artifacts that occur e.g. on the side of the transport truck. Looks like the contributions from the neighboring pixels need to be weighted adaptively or something?

It is obvious that (with 6x3) there must be a "type A" and "type B" of a sub-sampling matrix (because the Bayer pattern cell size doesn't fit a single sub-sampling matrix). So, we better know how Pentax has filled in the holes in the information they got from the sensor.

BTW, defringing is still probably best done in 768x1024 half images as there, fringes are 1px wide only.

I am now also thinking how one could measeure the position-dependent PSF in video mode (if Pentax doesn't release this info).

Any further ideas?

I tried processing the green-magenta fringing on a half image and got similar results compared to the full image. The Sobel operator is only a first-order approximation, so it doesn't necessarily represent pixel-accurate edges -- might need to shift the thing it to line up with the original image or work on an upsampled copy.

As for the other false colors in the street video, I think we'd need more information about how the images are demosaiced, like you were suggesting.

Could try simulating some demosaic algorithms with different sub-sampling matrices? Figure out how they handle gradients at a various angles/widths/colors, then compare with the actual camera's output?

[nulla]

I just tried Microsoft's new search engine Bing for K-7 videos and it works much better than Google/youtube . Just move the curser over the video thumbnail and it plays the video then just click to run it in normal view.
Pentax K-7 - Bing Video

Thanks for the Bing link... looks promising


cheers


Neil

[falconeye]

Do you think this is good enough to invest more time?

TBH I'm probably not the one to ask about that . It does improve the aliasing pretty well in a lot of places.

Hmmmh,

no further reactions. Seems like we now have lost the rest of this thread


Is anybody reading, still?
Shall paene and I continue to research ways to improve the video quality?


(Since I currently have a K-7 sitting on my desk, I would have to do this soon )