I have a collection of M lenses. Using them with the green button is a bit of an acquired taste compared to A lenses, never mind auto focus lenses.
It seems that using M42 lenses is a slower more deliberate process. In some cases such as landscapes or portraits this not necessarily a bad thing as it forces one to really think things through.
In the case of the fifties it is a tough call. Faster aperture versus faster handling.
Apart from the above it is down to IQ. As a lens hoarder myself, I would say keep them all.