[rawr]

Stochastic resonance

As soon as I saw that, I knew I had to give a like to your post.

PF, despite all it's sometime drama, is a great learning environment.

[Rondec]

I think Stochastic Resonance would be a great name for a band.

[Class A]

We seem to disagree that:
anything done in-camera can be done in post processing

We don't disagree on this point.

It is obviously true that there are noise reduction approaches that can only be performed in-camera.

Where we disagree is on our estimation whether this happens in the case of the K-1 II. According to the data I cared to assemble, it is very unlikely that the accelerator makes use of any information that either isn't already available in standard RAW files or could be made available (by adding some additional data).

signal can't be distinguished from added noise

With all due respect, I think your idea of signal vs noise seems to be based on the notion of a "strong signal" with identifiable features against a "weak noise" floor and in my estimation such a perspective is too naive for high-ISO (read "low light") scenarios.

We may also want to distinguish between "source noise" (Poisson distribution of photon events) which one could consider to be part of the signal vs system-generated noise (from analog amplification, A/D conversion, etc). How do you want to separate the "source noise" from the "system noise" (unless you gain access to sensor internals which I understand to be off-limits to the accelerator chip)?

It is too bad that member falconeye has defected to Nikon as he had some interesting thoughts about forgoing to even store all the "source noise" and then add it later (thus gaining more data compression in storage). This could suggest that there are ways to separate noise from signal that are not harmful, but I'm not immersed enough in this subject to competently comment. My current level of involvement with the matter suggests to me that you'll find it extremely difficult to separate weak signal from strong noise.

the chip is doing a "nearest neighbour"-type denoising approach

I'm not saying it is doing just that. I fully expect it to do something much more clever overall. However, the attenuation of high frequencies one can observe is consistent with the notion that some "nearest neighbour"-processing is part of the overall manipulation.

Actually, if you look carefully, the FFTs are not consistent with this at all.

I did spot features in the FT plots (FFT= Fast Fourier Transform / FT = Fourier Transform: The first "F" refers to an implementation choice which is irrelevant for the outcome) that are not consistent with a simple pure "nearest neighbour" processing. That is absolutely correct, but I never stated that the processing just amounted to a simple "nearest neighbour" smoothing. What I always meant to communicate is that a large part of the FT plot features is consistent with a "nearest neighbour"-type processing component.

In any event, to a RAW purist (who has good reasons for their principles, other than being pedantic) any kind of processing that isn't reversible is a problem; we need not fight over the specifics of the processing as long as we agree that it is destructive in a sense. I'm not sure you'd agree to that, but I don't know how you can explain why the 2D FT plots show high frequency attenuation while not destroying information. I'm making -- the very reasonable -- assumption that the processing cannot distinguish between noise and low, almost random signal, and thus would destroy signal as well as noise.

Noise has independent differences that are statistically bounded to modest values. Image detail has structured differences (and similarities) and are potentially unbounded in value.

I believe this is an example of where your view on signal and noise may be too simplistic.

Have a look at this visual demonstration of signal surviving noise levels that exceeds signal noise. Interestingly, the blog entry makes reference to our friends at DPReview who apparently espoused the untenable notion of signal only being meaningful as long as SNR >= 1.

Note that the "signal" in the examples chosen is pretty regular and thus nurture your hopes of extracting it by looking at pixel correlation, but in general the signal could be more random and I dispute the notion that any signal that looks like noise is noise.

As much as I totally respect MJKoski's photographic skills and applaud his efforts to push cameras to their limits, I'm not confident of his conclusions because I fear that his choices camera settings and the RAW developer may be contributing to the problem.

Possibly true.

A thorough analysis of the problem would have to go beyond anything that has been done by anyone so far.

I like this test!

Thanks!

Do you think I should propose it elsewhere more prominently so that there is some hope that someone may conduct the respective experiment?

What's interesting is that the phenomena of Stochastic resonance - Wikipedia implies that noise will actually help occasionally reveal the smoothed low-amplitude signal in some images of the stack.

I'd say that is only remotely relevant to the discussion at hand, as stochastic resonance is about passing thresholds that otherwise would block signal entirely. As such it seems related to the idea of "dithering" that is used in audio applications to avoid systematic quantization errors. However, a possible "take away" from this area is that there is "good noise", i.e. that by removing noise at the wrong stage or in the wrong way, you can introduce non-linearities that give rise to a bothersome artefacts that some people refer to as a "digital signature".

[MJKoski]

his choices camera settings and the RAW developer

Would you please tell me what camera settings were wrong? RAW developer on the other hand...if modern digital camera does not work with Adobe camera RAW or there is lack of understanding from manufacturer to get the support done in time, the whole system becomes dead fish in the water. One gets in bad position with Adobe, one vanishes from the market. Alternative options plain suck in user experience these days or fail to produce understandable archival solutions. Third option are products like Luminar which try to produce maximum bling-blong effects for social media like hunting (fail).

Whole digital camera thing is going straight to hell. More and more automation and soon camera suggests composition based on 100 000 other pics from the same scene. Photography does not exist anymore in digital domain in few decades. So better get ready to return to the roots.

[Class A]

if modern digital camera does not work with Adobe camera RAW or there is lack of understanding from manufacturer to get the support done in time, the whole system becomes dead fish in the water.

Apparently you expect optimal support for the K-1 II from Adobe already?

Even if Ricoh approached Adobe early on about the best support for the K-1 II then I don't think Adobe would have optimal support available already. Adobe does not properly support Pixel Shift to this day. Does that mean Pixel Shift becomes irrelevant? Just because one big player with their ransom-ware and "have all your auto-adaptively processed images on our cloud"-approach does not fully support Pentax?

It seems you are suggesting that we ought to blame Ricoh for Adobe's shortcomings.

I see where you are coming from, but you are taking it way too far.

Alternative options plain suck in user experience these days or fail to produce understandable archival solutions.

Capture One Pro is a perfectly fine alternative and I'm sure there are others.

If the shoddy stuff that Adobe calls "software" were the only recourse we had, we'd indeed be in trouble.

So better get ready to return to the roots.

I think many of us think you should just do that and leave us alone with our digital/Pentax troubles. We won't be mad with you, if you abandon us.

I appreciate it when you point the finger to something real (it isn't always real what you are pointing to, but sometimes you do have a point) but your negativity and propensity to boundless exaggerations are really quite tiresome. Instead of viewing yourself as being outside of this community (quote "You folks poison yourselves.") why don't you become a part of it and try to be more constructive?

[Kunzite]

RAW developer on the other hand...if modern digital camera does not work with Adobe camera RAW or there is lack of understanding from manufacturer to get the support done in time

It's the other way around - Adobe Camera RAW must work with the modern digital cameras.
You don't restrict the camera to work with a certain software; you adapt the software to support the camera.

Nice attempt at pushing Adobe's responsibility onto Ricoh Imaging.

---------- Post added 13-05-18 at 01:57 PM ----------

I appreciate it when you point the finger to something real (it isn't always real what you are pointing to, but sometimes you do have a point) but your negativity and propensity to boundless exaggerations are really quite tiresome. Instead of viewing yourself as being outside of this community (quote "You folks poison yourselves.") why don't you become a part of it and try to be more constructive?

+1

It's amazing how someone whose posts contain mostly poison can say something like that.

[MJKoski]

You do realize what I meant with that poisoning? World is not turning around Ricoh/Pentax and probably will not do so in the future. Why lose sleep over DPR article if you know it contains pure herecy? It seems to me it hit the nail to the head reading this topic.

Camera is just a lifeless tool to me. I repaired my primary camera with duct tape and glue last summer. It is made of wood and floats in the water. It has many failings and cannot be operated in the dusk or at night. It does not produce tack sharp image which is my requirement for digital device. It is only right that all devices violating the only holy grail of digital imaging get a severe beating. Period.

[Kunzite]

What you call poisoning I call nausea caused by poisoning.
There are small pieces of poison in your last post: implying that the world is turning around Ricoh/Pentax for us, implying that for us it's a (pseudo)religious matter; basically calling us zealots without pronouncing the word. This is no longer poison directed at our choice, but at ourselves - a community you're distancing yourself from. Sadly, for you we are the Others.

The second paragraph basically tells me that your camera is a witch (and what do we do with the witches? We burn them!)

[angerdan]

Just two comparisions that shows, what the K-1 II is capable of.

[photoptimist]

Would you please tell me what camera settings were wrong? RAW developer on the other hand...if modern digital camera does not work with Adobe camera RAW or there is lack of understanding from manufacturer to get the support done in time, the whole system becomes dead fish in the water. One gets in bad position with Adobe, one vanishes from the market. Alternative options plain suck in user experience these days or fail to produce understandable archival solutions. Third option are products like Luminar which try to produce maximum bling-blong effects for social media like hunting (fail).

Whole digital camera thing is going straight to hell. More and more automation and soon camera suggests composition based on 100 000 other pics from the same scene. Photography does not exist anymore in digital domain in few decades. So better get ready to return to the roots.

It looks to me like the anti-aliasing simulator is turned-off or set to the low value in some of your photos. There's Bayer CFA color aliasing in a lot of the hair. (My apologies if I'm wrong).

As for the RAW developer, those seem to have evolved from simple interpolating demosaicers to much more complex algorithms that attempt to guesstimate color from the noisy, CFA-aliased data. The advanced RAW developers are trying to make pretty pictures. Who knows what effects these complex algorithms might have on images from a camera that has something like the accelerator chip.

Has anyone ever tried to hack the EXIF of a RAW file to change the ISO? If changing the EXIF ISO changes the developed TIFF, then it suggests that the RAW developer is using the expected noise level associated with the EXIF ISO value to adjust how it deals with the image. That is, the RAW developer may have some denoising effects. But if those denoising effects are not properly tuned to the reduced-noise properties of K-1ii images, they may chew into image detail.

[MJKoski]

My observations have come to conclusion that not many are satisfied with mk2 as it is. People are afraid to speak of it. This is seen in hostility and doubt against all reports focusing on the accelerator unit, need to defend every last bit of "enhancement" of pre-war Safox AF unit and weird half-*peep* semi-PS routine which takes 20 seconds to write an image to a card implifying lack of processing power.

Technical IQ analysis by bclaff already proved that even with the accelerator one cannot boost the old horse to beat the fresh one. Not impressive result. It might be if Ricoh had the 42MP chip and managed to beat Sony in their own race.

What comes to Adobe and Pentax...

1) It still gives superb details out of any alternative with non-MC PS file
2) How can Adobe suddenly work out some propellerhead RAW (PS) which contains god-knows what? Ricoh needs to be the active player should support come in time. True, MK2 does not seem so hard provided that simple EXIF hack lets one work PEF. Unless, some workaround is needed because accelerator unit requires such.

Correct result of that DPR review should be something like 50/100 and copper award. Even Sony managed to improve their mk2 and mk3 models in huge steps in 2 years of time even if they seldom realize what functions a camera should have. Gosh, it took them 3 generations before they realized that an electronic device requires power to operate.

Ricoh is fighting multiformat war and lacking resources to back it up. How long does it take to get one 50mm lens to shops? Nikon just had to do something to change tactics.

@photoptimist:

Yes I will never use blur filter if camera has option for such. So all AA always disabled. That pixel-shift PEF I shared does not have bayer-problems and still suffers from noise reduction effects. I can try changing the ISO with Exiftool, or anyone can now.

[Tan68]

... I think many of us think you should just do that and leave us alone with our digital/Pentax troubles. We won't be mad with you, if you abandon us.

I appreciate it when you point the finger to something real (it isn't always real what you are pointing to, but sometimes you do have a point) but your negativity and propensity to boundless exaggerations are really quite tiresome. Instead of viewing yourself as being outside of this community (quote "You folks poison yourselves.&quot why don't you become a part of it and try to be more constructive?

I enjoy the schooling MJKoski provides and it is a good thing to be introduced to new ideas in an entertaining way. Particularly for adults, it is important to incorporate humor with learning; older folks aren't so used to learning new stuff and humor helps. Setting examples in context that is relevant to the audience helps people better relate to new infos. and this increases rate of information retention.

Most important for retention, though, is to periodically revisit topics. Following training, multiple short contacts to reinforce the new infos are best practice both to ensure information is retained and, most importantly, that the infos retained are correct.

I shouldn't judge negativity, exaggeration or constructiveness but I do think he uses an effective model for disseminating his information.

---------- Post added 05-13-18 at 08:08 AM ----------

...

Camera is just a lifeless tool to me. ... It is only right that all devices violating the only holy grail of digital imaging get a severe beating. Period.

And it is only humane that the object be unconscious, if not dead, during even a moderate beating.

[D1N0]

Another comparison with standard Pentax sharpening and NR applied (instead of dpreviews no software nr or sharpening approach) iso 51200:



What detail is lost here?

[Tan68]

Just two comparisions that shows, what the K-1 II is capable of.

Hello!

It is surely due to different lighting, but I think the hair looks more natural in the K-1 II sample.

I have decided the K-1 II does a good job with these samples.

Another topic > The DPR DR widget with both K-1 cameras exposed at ISO 100 and then lifted 6 stops does show the K-1 II less tolerant of this manipulation...

I am not someone who runs around grossly underexposing stuff only to later lift it several stops.. I do sometimes take single image 'hdr' exposures and these might be interiors with windows or sunrise or etc. I don't tone map but haven't seen much need for grad-ND since K-5

Anyway, I asked what people thought about the results of this DR widget in another thread but it... got locked.

I have since asked Bill Claff about this. He tolerated the slight highjack of his thread and replied about differences in color filter array. So I don't mangle his reply, see here : Help Sought Collecting Further PEFs for K-1 II Analysis at PhotonsToPhotos: Pentax SLR Talk Forum: Digital Photography Review

He notes it is not possible to draw generalizations so I take this to mean :
- The CFA on the K-1 II may be no different than that on tested K-1
- But variation and some other variables may lead to the less pleasing K-1 II result.

I have since asked two follow-up questions.

[Class A]

What detail is lost here?

Do you not see it?

Squint your eyes (to blur the images a bit) and you'll see structure in all of the coins of the K-1 image but very little in the K-1 II shots.

Now I could assume that the mushy mess by the K-1 II is due to the "accelerator" processing, but then I'd be crediting DPReview with having produced images that allow such conclusions. As it is, I don't know whether the loss of detail is due to the different lens or some other factor they haven't kept constant during the comparison.

Thanks, BTW, for the attempt to provide a level playing field with respect to the noise. I think you have the right idea, but as your source material has not been produced with rigour, I don't see how it can help to arrive at conclusions.