K-x annoyances: 1) The grip is just a few millimeters too short to make my little finger comfortable, 2) RAW files write SLOWLY on all affordable SD cards, and 3) it's so loud!
40D annoyances: too many to list, I really only liked the speed of it. If I were purchasing again I would just suck it up and get a 50D, which would still leave my ergonomics complaints and irritation with the lack of customization options, while bringing the file quality up to acceptable levels and providing a screen that allows decent preview.
K20D annoyances: operation was generally slow, shadows were noisy at all ISOs. Otherwise great camera.
E-330 annoyances (really digging back here!): small dim viewfinder, some half-baked functions, low resolution with high noise (though base ISO looked great if you didn't enlarge past 11x14 or so), even slower AF than the K20D, it was the last Oly shipped with the 14-45mm kit lens, which is one of only two Digital Zuiko lenses in existence that is actually best described as "mediocre," as far as I can tell (the newer 14-42 is fantastic). Wasn't even up to the demands of a 7mp sensor...
Despite all that, I've loved all the cameras I've owned and used (ok, I was a little more lukewarm towards the 40D, but I loved it for the 85mm lens I had mounted on it for the duration
). Nothing's perfect, and the closer you get to that unachievable perfection, the more things cost.