I basically ignore people when I am out shooting, if they are looking at me I walk right past them. If they ask me if I photographed them I just say, "Why would I photograph you, are you famous?" then show them all the photos I've taken of screws, signs and weathering on the concrete.
OK, Seriously this is my mindset but I'm actually pretty polite in person and I'll freely show any photos I've taken for reassurance. No need to add to the stigma. One thing I find works a treat is get MORE visible like a fluorescent vest so it looks all official.