I've often traveled with my da limited primes. In fact, it's my default because it's easier to pack given my camera and travel bags, unless I know I'll want a zoom. The other reason is that a zoom like the 16-85 is awkward to walk with if extended, the overall size of the lens even for a normal angle of view grows a lot in addition to the initial bulky size; the built in hoods of the limiteds make it even easier to have them around the city. Last trip was 15+21+40+70, other times I just take three, or two, or just one

So I usually start with either the 21 or the 40, then change as needed.
As long as I have the camera on a strap, changing lenses out of my bag while standing hasn't been an issue, but I wouldn't want someone to bump into me while I'm doing it so I try to go to a less crowded place.