May have been said above, but the set screws for the focus fine adjustment are coming loose (one is out already apparently) so your fine adjustment is drifting, probably every time you turn it against one stop or the other. I could tell you how to get at them on the identical A lens but not the M.
On the A you pop the front trim ring off and then you can see 3 tiny screws (JIS though they look like phillips they are not) to set fine adjustment you loosen them all and then while holding the outer focus right against the infinity stop, put the screwdriver into one of the screws and turn the whole assembly that the screws go into without letting the focus ring turn (like using a rotary phone with a pencil sorta
) either right or left to correct the focus at infinity then tighten the screws. Basically the screws hold the inside of the lens with the glass tight to the outside focus ring that you turn which has the stops on it, so its slipped and the outer part is hitting the stop before the inside reaches the infinity end of its rotation. I would avoid locktite though some suggest it, these things are tiny and super easy to snap off and with locktite on if you ever need to take them off again they will break. Have to just use finger tips tightening them actually and be super delicate. Also get JIS screwdriver set off amazon or something.
EDIT: just looked at that link, its the part in step 3 I believe though these things are definitely different inside. Not sure what he is talking about on his step 11, but it sounds more technically correct than the "looks correct" way I did it.