This thread highlights a challenge in comparing autofocus performance of different equipment and brands – a scarcity of credible quantitative tests and objective reports.
Anecdotes and hyperbole. In many reviews and discussions here and elsewhere, most comments on AF tend to be qualitative and anecdotal, whether positive or negative. We read statements along the lines of “the other guy got more in-focus keepers; my Pentax AF couldn’t compete,” “Brand X sweeps the floor with Brand Y,” and “Camera X has faster AF; I made a quick test.” While such sweeping statements probably reflect the users’ subjective
impressions, which shouldn’t be discounted, I would suggest that they shouldn’t serve as a basis for others to draw definitive conclusions.
Methodical quantitative tests and analyses are rare. The online reviews that purport to present credible tests typically fail to account for key parameters; they don’t describe all of their pertinent test conditions; or they don’t report meaningful data. Nor is it obvious that the tests have been performed with optimal camera settings or that comparable lenses have been used.
The Wobbly Bike. One of the more prominent series of AF tests conducted by DP Review – the so-called ‘
wobbly bicycle’ test – should be noted at least for DPR's effort to include it as a common reference point in their camera reviews. However, even this test has been conducted in various settings under uncontrolled lighting conditions, different riders wearing different clothing, and different motion dynamics, amongst other variations. DPR’s reports typically omit information that would allow the tests to be repeated; hence the tests are not entirely credible.
The scarcity of credible AF tests should not be surprising – these are difficult to set up and time consuming to conduct. Several key performance measures need to be considered, including time to achieve AF lock, accuracy, precision, tracking ability against several relative target motions, and distances to the target. These issues have been discussed frequently in other PF threads, and summarized succinctly by member @beholder3 here at post #6:
Accurate AF Lenses? - PentaxForums.com What about the Real World? Another gap is that anecdotes and tests are usually not extrapolated credibly to real-world in-field situations. For example, it might be claimed that Camera X’s AF is tens of milliseconds faster than Camera Y, or it achieves a few more in-focus shots in a long burst. The crucial question is how the performance measures translate to real-world situations – for example, are tens of milliseconds
really critical, or do nine in-focus shots in a burst produce a better
end outcome than seven?
A recent thread pointed to an AF test conducted by a reputable German magazine (also posted by @beholder3:
One of very, very few repeatable AF.C autofocus tracking tests - PentaxForums.com). Although the test was relatively narrow in scope, it showed how AF performance can be quite variable across brands and camera models depending on the conditions. The test could also serve to challenge those sweeping subjective claims that we hear frequently.
While we often hear the recurring theme that Pentax AF is inferior to other brands, one might wonder whether such sentiment is well founded, or based on limited personal experience, or simply a regurgitation of a popular myth that has not been updated to reflect contemporary camera systems.
- Craig