Remember that, in theory, aperture area is halved each time you close down the aperture by oen stop. HOWEVER, do the math yourself and you will see that the stops numbers are not Square root of 2 = 1.44 times larger or smaller than the stop before or after. They are an approximation.
What does that mean? It means that each lens can be tuned ever so slightly differently when you close down the aperture. Hence the differences (yeah, I used "hence
Also, you seem to assume that your prime is the culprit. Why not your zoom? It's much more likely to cause these kinds of problems, and it could very well vary with focal length.
Finally, take a lens with an aperture ring and play with it while looking at the blades. You will see that the difference between, say, f11 and f16 is very, very small. So at those small apertures it might be harder to tune correctly.