If you go the (mostly) prime route, the important thing (in my opinion) is to decide what your primary lens should be; the one that lives most of the time on your camera. For me that lens is the FA31Ltd, for you it can be a 24mm or a 40mm or ... The FA31Ltd is very universal and basically lives all the time on my camera (landscape, city, parties, whatever).
What suites you is up to you. You have the gear to test it. Set your 16-50 on 31mm for a couple of days and see if you can work (under different circumstances) with the field of view that it gives. Same for the decision at the long end; set your 50-135 on 77mm and see if it works for you. If not, try other focal lengths.
In both cases: no cheating
I consider the other lenses 'specialist' lenses. I think this allows me to limit the collection of what I need as well as the lens swapping.
With that in mind I added the DFA100WR to be able to zoom in a little on details and pull things a little closer; slightly longer than the FA77Ltd, macro functionality and reasonable suitable for (the occasional) portraits.
And one day I will need something wider (mostly for indoors in cathedrals and other buildings); I haven't taken my decision on that one yet, strongly thinking about DA12-24 and not a prime, mainly because my favorite field of view at the wide side is not available as an AF prime but also some flexibilty there.