Obviously if you record images of someone who doesn't want their photo taken then you're doing it wrong. If you keep recording you're really doing it wrong (and perhaps in danger of violent response). If you then market this footage commercially you're looking for a lawsuit.
I really cannot stand having my photo taken.. much more video.. esp with the ease of taking and the ease of sharing it online, a lot of stupid stuff happens.
So I try to go out of my way to respect people's privacy when I'm out shooting.. mostly at public spots.. parks, lakes, etc. Most people are pretty cordial but sometimes I get the unfriendly don't trust you look. I think the 'big' camera intimidates and scares some people... they want to know 'what is their intention with all of that hardware?'
And, saying that, I've had crazy run ins before while minding my own business.. such as when I was photographing flowers in a quiet, public park with a macro lens (really close to the flowers). And with my back turned, maybe 100 feet away from a playground area with a couple kids on it, never aiming the camera at the kids. Some woman in a truck on the exit road nearby to me slams her truck's brakes and slides to a halt, yelled angrily at me something about kids in the park and I shouldn't be here (I'm assuming because of the camera), and then she spun her wheels in a burn out and flew out of there.. she was livid. Until that moment I didn't even know she existed.
I finished my photos but didn't stay around long.. mostly because I had no idea where she was going and if, who, or what she was bringing back.. someone that 'emotional' is not thinking clearly.
Can you imagine if you were out in a public spot and you took HER photograph and she noticed you taking it?

Or you caught her kids in your shot?
Interestingly, I've yet to be stopped or questioned by the police out in town.. even setup with a tripod on a sidewalk along a fairly busy road (trying long exposures). They have to see me.. but they've been reasonable. No Barney Fifes thus far.. though I know they exist.