Glad to hear you've decided to shelf the honking 50-135mm. For me, the enjoyment I get from travel is directly linked to the size (or lack of) and simplicity of my luggage ... but of course everyone is different. Personally I'd take only the 20-40. That's it. Small, light, and unobtrusive, and fairly fast at the wide end where you need it most for darker interiors like churches and such. If absolutely necessary you could pocket the 70mm limited and still come in lighter than the 16-85. The 16-85 is similar to my Sigma 17-50 in size and weight. Excellent for a couple of days of shooting, but after walking around for a couple of weeks with it hanging off my shoulder and neck, waving around the huge front element/hood, it gets old fast ... just my opinion of course.
...as for traveling with a fisheye lens, its not my cup of tea. While its fun to have a handful of fishy images from a vacation, looking at an entire vacations worth of fishy images becomes a bit ... dare I say it - gimmicky
. If a wide angle lens is important, a rectilinear like your 14mm (or 15mm, may yours R.I.P) is in my opinion much more useful because you can shoot it all day every day and the effect never gets old. I bring my fisheye when I travel in my own vehicle and don't have to carry it daily, otherwise it stays home ... once again, this is just opinion and I'm in no way slagging others who like to travel with their fisheye.
- its interesting to see how divided the responses are - what to bring traveling is such a subjective subject!