I have a few different ways to deal with this too.
For mountain biking I put the camera in a Zing neoprene case or a small OR dry bag and stash it in my mountain biking pack. Access isn't as quick as other methods, but I can't have anything swinging around when I'm mountain biking. Too dangerous for me and the gear!
If I'm on foot or riding a bike around town I use a Black Rapid RS Sport strap. I like the little stops the strap has to control where the camera hangs. Works well.
If I'm skiing or flat water kayaking I use a Clik Elite chest pack with bigger lenses, or the chest pocket of my parka if I'm just bringing a small prime.
Here's me with the chest pack: