I always keep a lens cap on, although I may keep it off if I'm shooting a lot of pictures in sequence. The more it stays on, the more dust, grime, and funk stays off the lens (or filter), so the less cleaning you have to do. I try to clean my lenses as rarely as humanly possible, so I don't scratch them. For some reason I am paranoid about such things, even though rationally I know that lenses are MUCH more scratch-resistant than most lens owners make them out to be. It is actually very difficult to scratch a lens under normal circumstances.
Of course, it is those out-of-the-ordinary circumstances you have to watch out for...