[beholder3]

The German photo magazine "C't Digitale Fotografie" in issue 02/2016 had the only autofocus test procedure I have ever (!) seen which is repeatable, well documented and to be taken seriously.
Quite a contrast to the clown show at dpreview.

I just provide the results here to provide some facts versus the fanboy claims of "brand x nails every shot" level.

The magazine spent two full pages to document their test setup and methodology and measuring, which I am far to lazy to repeat here.
The shot about 3,900 images in total!

Setup in short:

• They set up a white beamer canvas against which a beamer projected images of a small flash program which displayed either a swallow or a female face and these were moving at a predefined speed across the image.
• The camera was sideways to the canvas and shot the image from a 30 degree angle (the subject images were electronically distorted to provide a normal look from 30 degrees).

3 Test scenarios for all cameras:

1. Tracking
   The camera on a tripod was zoomed to cover the whole image area with AF points. A bird (black and white with extra contrast marker) and also a face (much softer contrasts) image flew across the canvas from edge to edge (remember the edges are in different distances). The lens was manually set to middle distance at start.
2. Follow focus center AF point
   Same bird, but now the camera was set on a swivel panoramic head and a user turned the camera to keep the subject under the AF point. So the camera just had to adjust z-axis
3. Focus with obstacle
   A little real life plant was set in the middle between the canvas and the camera. A face moved from left to right over the screen (thus was hidden briefly by the plant), while the camera was panning, following the face. Just the type of scenario you set your Pentax AF hold parameter to "high" for.

"Sharpness" evaluation criteria:
They did not even rely on human arbitrary judgementto call an image "sharp", they used an automated analysis of the structured canvas and beamer signal creating moire where the focus area was sufficiently sharp (they looked at contrast curves. I can not really explain how they look, but it is extremely obvious where the focus was) . Their criteria were possible due to the fact that by using the angled canvas approach their target subject always showed some depth and it was not just a flat object which appeared sharp or not.

• "sharp" = target subject fully in focus
• "mediocre" at least half of the target subject in focus area (like the back wheel of a bicycle)
• "unacceptable" = target out of focus

Results:

1. Not one camera even came close to "nailed every shot". Let alone one brand.
   The authors declared Canon cameras to be in the lead by a small margin. They were completely disappointed from the Sony.
2. Hugely lens dependant
3. Practically nothing there where one can generalize from. Hugely dependent on exact camera/lens/scenario combination.

Anyone could redo the exact same test as it is well documented and the subjects are computer generated. You just need a flash program or a video of the target. I do consider the setup to be rather smart. You can redo that in your large garage.
Obviously the range difference obtainable is limited by the size of the canvas, so you can not simulate that fighter plane directly coming at you from infinity. But given the soso results this setup is interesting enough.

This is a test that comes close to scientific requirements and standards.

---

Last edited by beholder3; 07-08-2016 at 01:19 PM.

[grahame]

Thanks for sharing!
I recently helped a Nikon shooter fine adjusting her D800 and her lenses. She had her camera for quite a while and never thought of AF error, but many of her photos were out of focus.
Long story short, it turns out many of her lenses need adjustment, just like my lenses on K5. And during the testing and adjustment, I also noticed AF inconsistency that bothers me a lot on K5 and K5II. some lenses could focus spot on 3 times in 5 tests, but other 2 shoots were off. I guess many Pentaxian have experienced such thing. And now I can tell you, at least on this D800, things are not much better.
I think many pentaxians on this forum are sensitive or knowledgeable enough to notice the problem, and many Canikon users are not, like the lady I helped. She started with a super zoom on her D800 and never found serious AF problem (of course). But after she bought in more faster lenses, she complained better lenses gave her worse result, but never thought it was camera/lens issue - I guess partially because of the reputation of "nailed every shot".

[stevebrot]

Thanks for the very interesting post. The test methodology appears to be sound, but it is the results that blew me away. What I see in the tables reflects field experience and discussions I have had with various photographers over the years. Nobody is particularly happy with their AF and some (mostly Sony folk) simply shrug and say, "why bother".

I suspect that much of the emphasis on frame rate and buffer size is due to the need to "spray 'n pray" as remedy for poor AF performance.


Steve

[carolina_sky]

What this tells me: AF is a vastly complicated area with a combination of Camera (hardware and software), Lens (hardware and software) decisions, tradeoffs, and bugs that all go in to the melting pot of perceived AF accuracy. There are a near infinite number of permutations of scenarios. The DPReview people picked on a particular scenario that they *knew* would foil Pentax AF and based their whole eval on that one scenario. Interestingly their IQ testing is far more rigorous and hence much more acceptable. Why they employ such rigor in one aspect and such half-assedness in another is beyond me. They also pretty heavily weighted their final conclusion on the one disappointing aspect.

Frankly, in the end I don't care. Haters gonna hate. I'm very happy with my K-1. I shoot both landscape and sports (horse shows). In landscape, the K-1 with pixel shift is phenomenal. My D800 toting friends are actually jealous (though the 800/810 is still a very fine camera in its own even though its going on 4 years old). In sports, the K-1 AF + 150-450 lens is giving me higher hit rates than ever before (I would guesstimate in the 80%-90% range) in some difficult situations. I can actually shoot with AF.C now (9 spot center focus, not single spot), where I never could before. Is it perfect? No. Neither are the 810, 7rii, 6D, etc.

I do feel bad for Pentax that DPReview gave them a slightly tarnished silver for what is beyond doubt the best value in full frame on the market. I sure hope it doesn't significantly affect their sales. I find much to commend in the review but the AF portion is completely amateurish. I don't care if they want to bitch about menus being too deep or other qualitative nitpicks. they are entitled to voice opinion on that. But AF is quantifiable and if they're going to be judging it then they should come up with a better way to *actually* quantify it. Thanks for the link to the AF study above. Too bad it doesn't include Pentax. Hopefully they do an update, this is the kind of more measured testing that I can appreciate.

[monochrome]

The only good news is Pentax sales in the USA are so abysmally low that this kerfluffle won't even register a blip on global success of K-1 for Ricoh.

[beholder3]

I just added some sentences on how sharpness was assessed.

[Dlanor Sekao]

Wow ! This chart Blew Me Away ! What a AF Novice like myself sees is ALL cameras pretty much Suck in autofocus quite a bit....thanks for sharing.

[gaweidert]

Wow ! This chart Blew Me Away ! What a AF Novice like myself sees is ALL cameras pretty much Suck in autofocus quite a bit....thanks for sharing.

That is why an old fart like me still wants optical focusing aids on the focus screen.

[Oakland Rob]

Crikey. That's almost as bad as me with my old film cameras. Doh.

The lens dependence is huge, though. While the K-1 itself is a good value, add in the full frame lenses and maybe not so much, especially as we see here for something like sports that wouldn't exactly help in AF performance. But as others have noted, the new mount and new lenses may make a big difference.

Thanks for the info; interesting.

[beholder3]

This well done test does serve well as a reliable credibility measure when you read some posters claims.

Probably little competence and credibility in other posts as well if you see an author who once claimed fanboy things like "nails every shot" or "not a single frame out of focus ever".

[wizofoz]

Thats an impressive test, unlike the DPR random biker.

[beholder3]

This is just to re-add the images that had gone missing from the starting post.


The German photo magazine "C't Digitale Fotografie" in issue 02/2016 had the only autofocus test procedure I have ever (!) seen which is repeatable, well documented and to be taken seriously.
Quite a contrast to the clown show at dpreview.

I just provide the results here to provide some facts versus the fanboy claims of "brand x nails every shot" level.

The magazine spent two full pages to document their test setup and methodology and measuring, which I am far to lazy to repeat here.
The shot about 3,900 images in total!

Setup in short:

• They set up a white beamer canvas against which a beamer projected images of a small flash program which displayed either a swallow or a female face and these were moving at a predefined speed across the image.
• The camera was sideways to the canvas and shot the image from a 30 degree angle (the subject images were electronically distorted to provide a normal look from 30 degrees).

3 Test scenarios for all cameras:

1. Tracking
   The camera on a tripod was zoomed to cover the whole image area with AF points. A bird (black and white with extra contrast marker) and also a face (much softer contrasts) image flew across the canvas from edge to edge (remember the edges are in different distances). The lens was manually set to middle distance at start.
2. Follow focus center AF point
   Same bird, but now the camera was set on a swivel panoramic head and a user turned the camera to keep the subject under the AF point. So the camera just had to adjust z-axis
3. Focus with obstacle
   A little real life plant was set in the middle between the canvas and the camera. A face moved from left to right over the screen (thus was hidden briefly by the plant), while the camera was panning, following the face. Just the type of scenario you set your Pentax AF hold parameter to "high" for.

"Sharpness" evaluation criteria:
They did not even rely on human arbitrary judgementto call an image "sharp", they used an automated analysis of the structured canvas and beamer signal creating moire where the focus area was sufficiently sharp (they looked at contrast curves. I can not really explain how they look, but it is extremely obvious where the focus was) . Their criteria were possible due to the fact that by using the angled canvas approach their target subject always showed some depth and it was not just a flat object which appeared sharp or not.

• "sharp" = target subject fully in focus
• "mediocre" at least half of the target subject in focus area (like the back wheel of a bicycle)
• "unacceptable" = target out of focus

Results:



and

1. Not one camera even came close to "nailed every shot". Let alone one brand.
   The authors declared Canon cameras to be in the lead by a small margin. They were completely disappointed from the Sony.
2. Hugely lens dependant
3. Practically nothing there where one can generalize from. Hugely dependent on exact camera/lens/scenario combination.

Anyone could redo the exact same test as it is well documented and the subjects are computer generated. You just need a flash program or a video of the target. I do consider the setup to be rather smart. You can redo that in your large garage.
Obviously the range difference obtainable is limited by the size of the canvas, so you can not simulate that fighter plane directly coming at you from infinity. But given the soso results this setup is interesting enough.

This is a test that comes close to scientific requirements and standards.

[zapp]

A flat object tracked ona canvas. Projected by beamer with a certain frrequency. This may sound repeatable, but are we sure that this setup resembles real life?
In our company we use a model train to check our tracking sensors. Much easier.

[Dan Rentea]

A flat object tracked ona canvas. Projected by beamer with a certain frrequency. This may sound repeatable, but are we sure that this setup resembles real life?
In our company we use a model train to check our tracking sensors. Much easier.

I have 6 user cases for Af-C and each case has 3 individual adjustments. I needed a few shooting sessions with different subjects in order to learn when I have to switch the entire Af case or only one or two adjustents out of three from each case. And after I learned how to deal with those 6 user cases, I jumped to the next adjustments: focus priority, release priority and how/when to use one or the other. Then there is the af point selection (one, 5, 9, iTR, etc.) which is also very important. And so on... If they didn't take the time to learn each system and how the af works on each system and they just relied on what they read in the user manual, then the test is not that acurate. But even so, it's a test that can increase the confidence of beginers in their cameras, which is a good thing.

[rawr]

Interesting variance in the Nikon D7200 vs D810 test results with the same lens and AF settings. Notable that D810 generally performed better.

Those two bodies share the same 51 point 'Multicam 3500' AF module, but D810 uses a 91k pixel RGB metering sensor vs the more modest 2k pixel metering sensor of the D7200. (D810 can also do face detection via PDAF, assisted by the metering sensor). It all seems to have helped D810 AF performance, particularly for the '3D Tracking' AF mode both cameras used in these tests, which depends on the RGB sensor providing colour info to discriminate and follow AF targets.

FWIW, the Pentax 86k pixel RGB metering sensor is there to perform a similar role in K-3/K-1/KP.