That's sounds bad and I am totally understand your frustration.
I had similar experiences in public places but never have anyone ask me to pay. Maybe I have never photographed a cow!?
The latest incident for me was early last month in front of a Train station which I know for a fact that it is ok to photograph even inside a train station here, and people here do it all the time with either phone or camera. I walk past this place with interesting ceiling light with my tour group, so we stop taking a photo. Basically pointing cameras up for the ceiling, not even pointing at what in front. An old lady stops at a distance and starts yelling in Japanese." Why you guys taking a photo here. YOu, foreigner...(discrimination wording follow)...etc".
If you live here you will know discrimination comment is still fairly common, especially if you don't have a Caucasian look. Anyway, Not everyone does this. Just "SOME" and mostly older generation. [I can imagine seeing these people freaking out seeing mass foreigner coming for the 2020 Olympic, lol.] No-one in my group understands Japanese neither care to look at her. They were busy composing a shot. I am the only one not taking a photo. [I sometimes do a photo walk, and when I take people out, I usually watch out for an oncoming car, and or any potential weird people (like this old lady!).] I told my group to wrap up, and we leave. I explain to them what just happened later on.
I hate this type of non-written rule people made for themselves and try to enforce it on another even in a public space. But what can I do.