Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
03-09-2019, 10:11 AM - 3 Likes   #1
Pentaxian




Join Date: Dec 2011
Posts: 2,143
Understanding how foolish it is to make sweeping claims about autofocus

Often we read anecdotal claims about autofocus.
Sadly these anecdotal claims are no better than the anecdotal reasoning people give to support racism and human stereotypes.

Only a robust at least semi-scientific = absolutely repeatable test setup can actually prove things.

Sadly this is immensely complex and also a huge effort.

There is probably less than five autofocus tests which have ever been published around the world to even come close to credibility and reliability.

Here is one AF.C-Test from 2016:
https://www.pentaxforums.com/forums/190-pentax-k-1/325387-results-really-rep...ml#post4600445
https://shop.heise.de/katalog/autofokus-im-test


Here is one AF.C-Test from 2019:
https://www.pentaxforums.com/forums/38-photographic-technique/383586-general...ing-tests.html
https://www.heise.de/select/ct-foto/2019/2/1551446274805953




I guess most readers don’t grasp the complexity.
Here is a try to explain the minimum requirements for an „ok“test group:


Questions /qualities to evaluate:
  1. AF.S speed when camera has to wake up from standby + acquisition
  2. AF.S acquisition speed when already active
  3. AF.S precision for sharpness
  4. AF.S capability to not get distracted by background
  5. AF.S repeatability
  6. AF.S capability to choose tiny focus area
  7. AF.C speed to acquire first good shot
  8. AF.C speed to require focus after it was lost
  9. AF.C speed to predict and follow z-axis movement
  10. AF.C capability to not get distracted by backgrounds
  11. AF.C capability to track subject on x/y-axis
  12. AF.C capability to track subject on x/y-axis while distractions pass
  13. AF.C precision during a series
Variations:
  1. Body used (e.g. < $1000 / <$2000 / <$6000MSRP)
  2. Lens used; all need to cost roughly the same MSRP and should be roughly the same age ( <$1000 portrait prime / <$1000 tele prime / <$600 kit zoom /<$2000 tele zoom / <$2000 long tele zoom)
  3. Light conditions (EV 0/5/8/12/15)
Parameters to keep absolutely static:
  1. Exact same 3rd party lens to be used for evaluating the body performance
  2. Subject
  3. Subject facing
  4. All camera parameters (e.g. focus priority vs. Shutter priority; raw vs. Jpg; processing…)
This leads to the fact that you essentially would need to run at least 900 (!) test scenarios to get a reliable picture. Simply because you have 13 questions * 3 bodies * 5lenses * 5 light conditions.

And there are good reasons to differentiate this even more.

Even if you run one test scenario properly you only can validly describe 1/900th of the facts with some statistical relevancy.

Each of those test scenarios will require dozens if not hundreds of shots to be taken and evaluated.


Last edited by beholder3; 03-10-2019 at 01:22 AM.
03-09-2019, 10:26 AM   #2
Loyal Site Supporter
Loyal Site Supporter
monochrome's Avatar

Join Date: Mar 2008
Location: St. Louis, Missouri
Photos: Gallery | Albums
Posts: 23,936
To evaluate against a standard rather than just to make a relative set, you need an ageed scoring standard, the controllable, repeatable environment (constant and repeatable EV, subject and test) under which the scoring standard was created AND you need the agreed, ubiquitous control camera and lens to ‘prove’ the control standard for the tests. That way you can rank cameras against an agreed standard to show their ACTUAL differences in effectiveness rather than merely relative scores.

Last edited by monochrome; 03-09-2019 at 12:11 PM.
03-09-2019, 10:49 AM   #3
Loyal Site Supporter
Loyal Site Supporter




Join Date: Sep 2011
Location: Nelson B.C.
Posts: 3,663
Another test is steady/unsteady hand. I have a friend who has a slight Parkinson's tremor and shoots long. He gets amazing shots in spite of it.

This is an advantage of lens stabilization, or potentially.

The x/y movement would need a few iterations; one that fills half the frame, one that fills an eighth of the frame and one that is slightly larger than a focus point.

I'm going for 2000.
03-09-2019, 11:40 AM   #4
Pentaxian




Join Date: Jun 2015
Location: South West UK
Photos: Albums
Posts: 1,422
...or one extremely accomplished photographer who has used every body/lens combination extensively and is willing to write a very long report.

03-09-2019, 12:00 PM   #5
Pentaxian




Join Date: Dec 2011
Posts: 2,143
Original Poster
QuoteOriginally posted by victormeldrew Quote
...or one extremely accomplished photographer who has used every body/lens combination extensively and is willing to write a very long report.
Even that is irrelevant for the vast majority. Because relevancy would depend on the reader having very much the same usage patterns in detail.
This pretty much is being excluded automatically because of statistics.
03-09-2019, 12:16 PM   #6
Pentaxian
Site Supporter
normhead's Avatar

Join Date: Jun 2007
Location: Near Algonquin Park
Photos: Gallery | Albums
Posts: 31,917
QuoteOriginally posted by beholder3 Quote
Even that is irrelevant for the vast majority. Because relevancy would depend on the reader having very much the same usage patterns in detail.
This pretty much is being excluded automatically because of statistics.
And because every person has set of biases that may or may not be relevant to everyone. It is impossible to tell the difference between a person who says a feature is better because it's intuitive to him/her and therefore more enjoyable, or if the product is actually better. Depending on one person, means finding a person without bias. That simply is impossible. No such person exists.
03-09-2019, 12:40 PM - 1 Like   #7
Kiwi Pentaxian
Loyal Site Supporter
NZ_Ross's Avatar

Join Date: Mar 2012
Location: Timaru
Photos: Gallery | Albums
Posts: 3,491
QuoteOriginally posted by beholder3 Quote
Even if you run one test scenario properly you only can validly describe 1/900th of the facts with some statistical relevancy.
Some very good thinking there. Thanks for setting this out.

Our society has this very interesting paradox - on one hand we are at a peak (that is still progressing) of scientific discovery in a wide number of fields, and on the other we seem to have increasing scientific illiteracy around method, valid use of statistics and how validated repeatable testing actually works.
03-10-2019, 05:53 AM - 2 Likes   #8
Pentaxian
Lowell Goudge's Avatar

Join Date: Jan 2007
Location: Toronto
Photos: Gallery | Albums
Posts: 15,615
Given the obvious and daunting task of doing all the tests the op suggests, why not joust go out and shoot real world subjects. No professional photographer made any money shooting test targets for a living.

03-10-2019, 07:07 AM - 2 Likes   #9
Pentaxian
Site Supporter
normhead's Avatar

Join Date: Jun 2007
Location: Near Algonquin Park
Photos: Gallery | Albums
Posts: 31,917
QuoteOriginally posted by Lowell Goudge Quote
Given the obvious and daunting task of doing all the tests the op suggests, why not joust go out and shoot real world subjects. No professional photographer made any money shooting test targets for a living.
The only test that matters is "it works for me" .
03-10-2019, 07:29 AM   #10
Pentaxian
Wheatfield's Avatar

Join Date: Apr 2008
Location: The wheatfields of Canada
Photos: Albums
Posts: 11,009
QuoteOriginally posted by Lowell Goudge Quote
Given the obvious and daunting task of doing all the tests the op suggests, why not joust go out and shoot real world subjects. No professional photographer made any money shooting test targets for a living.
I think this is the most valid test of all. One may only come up with anecdotal evidence, but there is no reason why that isn’t valid. The idea that something has to be dissected to the Nth degree to make findings valid is pure horse patootie. The real world is where we take our pictures. I’ve never seen a picture of a test target win an award, or hang in a gallery, and in my many years as a working pro, I never had a customer order a picture of one to hang on his wall.

We don’t need to know to the millisecond how far behind our brand’s AF tracking is compared to the state of the art, for example, we only need to know that if that particular metric is the deal breaker, then Pentax is probably not the droid we are looking for, and we can base our decision on renting or borrowing gear from another brand and trying it out under our own shooting conditions.
If we get more keepers from another brand, do we really need to know to the millimeter how far behind the action one brand is compared to another, or is it enough to know that one brand works better for what we do?

I happen to shoot pretty static landscapes and still life images. AF is well down my list of do or die features. My needs are well addressed by Pentax as what the brand does well happens to fit my photographic lifestyle. I don’t really need to know with any greater precision how much better at what I do Pentax is over the other brands, and someone who shoots subject matter that Pentax is weak at really doesn’t need to know by how much, only that another brand fits his or her shooting style better.

Last edited by Wheatfield; 03-10-2019 at 08:33 AM.
03-10-2019, 08:39 AM   #11
Pentaxian




Join Date: Aug 2013
Location: Iloilo City
Photos: Gallery
Posts: 1,364
Measuring to the last millisecond will only establish which one is fastest. I believe it's an advantage to have a faster AF.C. But, it won't make that much difference if my camera can track the subject and capture the images correctly as a camera from another brand with the faster AF.C. The difference would be minimal. It sometimes turn out to be bragging rights.
03-10-2019, 08:46 AM - 1 Like   #12
Loyal Site Supporter
Loyal Site Supporter




Join Date: Sep 2011
Location: Nelson B.C.
Posts: 3,663
A K1 or K3 in my hand has far better autofocus than a D850 on a store shelf.
03-10-2019, 08:53 AM   #13
Pentaxian




Join Date: Dec 2011
Posts: 2,143
Original Poster
QuoteOriginally posted by normhead Quote
The only test that matters is "it works for me" .
+1

And this is the exact opposite of sweeping claims that "x is better than y", where x typically is a whole brand with a gazillion of product combinations and attributes.


"it works for me" = "I like vanilla better than chocolate" - grown up

versus
"vanilla sucks compared to chocolate" - child

The first line needs no measuring as it is just a minority personal opinion / preference.
The second is a claim on facts (which never are provided as evidence).
03-25-2019, 12:06 PM   #14
Senior Member




Join Date: May 2013
Posts: 122
I think it is naive to expect such an overblown test setup as beholder suggests?

Especially since most real life AF challenges can't be simulated using robotic test setups anyway.

A good photographer who had a lot of cameras in his hand for testing can make a good judgment nevertheless.

Let's remind ourselves on the K-1 AF deficiencies as identified by the two best and most diligent testers / tests that I'm aware of.
Probably they could and would be confirmed by an over-engineered test setup, yes, but that effort is actually not required, to come to meaningful conclusions about the K-1 AF, such as the following:

Pentax K-1 Review: Digital Photography Review
"don't seem to make that great of a difference in terms of performance as the K-1's autofocus system behaves in much the same manner as the K3 II. Even in the most basic, single point AF shooting modes, the results are far from what we would expect from a modern DSLR focusing system. The autofocus tends to hesitate, even in AF-S mode with the center point - meaning it's not as consistently fast as most Canon and Nikon DSLRs.

This hesitant behavior is more noticeable in AF-C mode, with focus falling behind the subject then having to jump to catch up. Subject tracking - where the camera shifts the AF point automatically to follow your in initial subject if it moves away from the initial AF point - has a very poor hit rate and seems to default back to infinity once focus is lost.

In addition to these limitations the focus points aren't illuminated in the viewfinder until you lock focus, at which point they glow red. This was a bit frustrating when attempting to focus in low lighting conditions. After all, if you don't know where the focus point is before you even initiate focus, how do you place your point over your subject properly?"

"single point continuous autofocus [...] even that failed about half of the time"

"Continuous AF with a single point really struggled to maintain focus on an approaching or receding subject - something DSLR AF systems tend to do really well."

"In addition to those issues the subject tracking mode seemed to have a great deal of trouble with even the slightest amount of movement."

"the subject tracking failed nearly 85% of the time with limited attempts to reacquire the subject after losing focus."

"The final point of frustration came when we reviewed the K-1's images"

"acquisition speeds in dark and low contrast shooting scenarios can be sluggish"

"AF-S too slow to keep up with the movement of the subjects, or to nail the decisive moment as soon as it happened"

"AF points do not illuminate until focus is confirmed. This is incredibly problematic"

"low light AF sensitivity [...] the Nikon [D750] did so significantly quicker and more reliably, which meant the difference between a focused and missed shot."

"[The D750] is a class leader in terms of low light AF speed and performance. Even a Canon 6D focused more quickly and confidently [than the K-1] in and around -2 to -3 EV (in AF-S)."



Taking on the DSLR Giants: Fstoppers Reviews the Pentax K-1 Camera | Fstoppers

"Autofocus - This is far and away the weakest part of the system. Unfortunately, the system is just not of the level we expect in 2016."
"33 points [...] is a little on the low side for a full-frame DSLR"

AF-S: "the hesitation is a bit more worrisome in continuous mode. Moreover, as lighting gets worse, the hesitation gets longer, often to an unacceptable point. Good AF point sensitivity doesn't mean much if the speed isn't reasonable."

AF-C:
"This is where things got frustrating."

"Horse shows are a great autofocus test [...] not overly difficult test. Unfortunately, out of about 480 images, 350 were entirely out of focus, 90 missed enough to be throwaways, and about 40 were acceptable. There didn't seem to be much consistency"

"This is frustrating"

"I could predictably tell where the camera would lose autofocus"

"It's a shame"

"I couldn't trust the camera's continuous autofocus"

Some stuff for Google translate:
DSLR-Forum - Einzelnen Beitrag anzeigen - Wie deutlich sind die Unterschiede zwischen der K-1 und der K-1 II beim Fotografieren?
"Der AF der K-1 ist eigentlich nur grottig. Er ist deutlich langsamer als die gelben Marktbegleiter [his Nikon D4 and D850]. Und er ist mir bei Dämmerung und Nacht nicht treffsicher genug. D.h. kleine Lichtpunkte die ich als Fokusreferenz nehme oder bei Dämmerlicht (frühe goldene und späte blaue Stunde), also da wo die Landschaftsfotografie gerne stattfindet, ist sehr oft überhaupt kein AF möglich."

Don't know what the slang "grottig" translates to best, it is probably like "abysmal".

To pack all of this know-how and experience into some formalised testing would be a huge effort, but would still come to the same conclusions. So nobody bothers really.
03-25-2019, 03:39 PM   #15
Pentaxian
c.a.m's Avatar

Join Date: Jan 2013
Location: Ottawa, Canada
Posts: 1,621
@Frater, I think you are correct -- such a detailed, complex and complete test would not likely be conducted, especially across a wide range of camera brands and systems. That would be a huge undertaking. In my mind, @beholder3 is outlining the complexity and difficulty of AF testing.

QuoteOriginally posted by Frater Quote
Let's remind ourselves on the K-1 AF deficiencies as identified by the two best and most diligent testers / tests that I'm aware of.
DP Review. It might be worthwhile to mention that the DP Review's testing of the K-1 -- in particular their AF assessment -- prompted criticism by Pentax user communities of DPR's testing approach and methodology. There are numerous discussions here at Pentax Forums on this issue, and comments at DPR also raised serious concerns.

Although DPR makes a reasonable effort to cover a large range of camera systems, their tests and reports have gaps. But that reinforces the point made in the OP above about the complexity of AF testing (and testing and comparing cameras in general).

Fstoppers. Concerning the second review at Fstoppers, I recall when it was published. I would suggest that the quality and credibility of that review should be questioned. For example, one of the strong statements is:

"What's frustrating is that there's little in the way of customizability in the menu system. Options like telling the camera to ignore closer subjects that suddenly pop into the frame aren't there. This is frustrating, as by the end of the derby, I could predictably tell where the camera would lose autofocus as I panned across a closer jump while following a rider and horse."

That situation is precisely the intent of the K-1's Hold AF function. It is described in the K-1 Operating Manual, and I think it's described briefly on Ricoh Imaging's Pentax site (maybe under K-1 autofocus features?). Although the description in the manual is cursory, I think the intent of that function should have been clear to the reviewer. Although the Fstopper review mentioned a number of positive points about the K-1, this important error brings the review into question in my opinion.

The third item in your post seems to come from a Forum thread. Would you happen to know the credibility of the commenter and the validity of the comment, or the background of that discussion?

To provide a counter-point to the two review articles mentioned above, the following piece by our own @beholder3 is illuminating. It uses horse jumping to explain AF functions, settings, and techniques. https://www.pentaxforums.com/articles/photo-articles/dslr-sports-autofocus-settings-guide.html.

In particular, Hold AF Status is emphasized: "The most crucial AF setting for show jumping is probably the "Hold AF status" setting which governs the reaction of the camera to things blocking sight onto the subject briefly or to situations where the user tracks the subject poorly and doesn't hold all AF points in use over the rider, but over some distant background"

- Craig

Last edited by c.a.m; 03-25-2019 at 05:54 PM.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
af.c, af.s, autofocus, camera, capability, claims, dslr, focus, photography, precision, subject, track
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
Is the K1 a foolish choice if you don't have any modern glass ? rjm007 Pentax K-1 77 08-24-2018 11:30 AM
Macro Sweeping Turn eaglem Post Your Photos! 2 06-27-2013 08:57 PM
Remembering 1989's sweeping wind of change causey General Talk 8 11-17-2009 08:11 AM
Feeling foolish and gulity.... madisonphotogrl Photographic Technique 25 10-08-2008 09:16 PM
Tomb Sweeping Day roentarre Post Your Photos! 21 04-07-2008 10:56 PM



All times are GMT -7. The time now is 11:32 PM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top