Some related musings (or thinking aloud
) ...
One aspect of BF/FF problems is that the (copy of the) lens used affects it: the very existence of lens (copy) specific compensation settings of the fancier DLSRs are proof enough of this IMO. The underlying reason is shrouded in mystery though (or at least I have not seen a convincing explanation ... probably a combination of a multitude of factors with a complex interaction which would make the explanation too hairy to be convincing
). The root cause (or one of them at least
) of BF/FF is different wavelengths of light having different refractive indexes (or propagation speed in mediums other than vacuum such as the optical glass); this is why/how a glass prism produces a spectrum of light visible to the naked eye. Lens makers put in much effort to compensate by clever use of materials with different refractive indexes and this is where things start to get seriously complicated
. That is, the good job that is apparenty done in getting rid of chromatic aberrations in the visible spectrum might come back to bite us here (?).
BF/FF stated in another way is that the focused image the AF sensor "sees" is different than what the image sensor "sees". A number of factors is at play, among them different spectral sensitivity of the AF sensor light detectors to the image sensor light detectors, the AF sensor "seeing" only a strip of the image (or a narrow section of the scene), physical mismatch of the AF/image sensor planes, and mechanical limits to the precision/repeatability adjusting the focus mechanism (as in a bit of simple mechanical play and inertia affecting things).
I suppose the firmaware / AF algorithm can compensate the effects of essentially physical imperfections to some degree, most likely at the cost of AF speed. Say, a simple, fast algorithm might do just fine when everything happens to be in the middle of its respective tolerance, whereas some other combinations might be overcome by tweaking the algorithm to compensate which is likely to require more data being collected as in allowing for longer integration times and more readings of the AF sensor while adjusting the focus mechanism. An outright bug (say, register/variable overflow) in the AF algorithm implementation can throw things out of whack, but it can also be about the algorithm being in need of improvement / tweaking: with this in mind the line between a physical / software defect becomes blurred.
Image sensors having gotten ever more less noisy has pushed the practical use of AF into lower levels of light where this is difficult to do / new, also, the ready ability to pixel peep also means the ability to see focus errors much more clearly.