The difference can be *huge* in the situations where it matters, but it doesn't always matter. Obviously, your test happened to be of a case where it didn't.
I'd say if you're sufficiently expert to be able to predict in advance when it will be needed and when it won't, and have plenty of time to waste constantly putting your hood on and taking it off, then by all means, only put it on when needed. For the rest of us, it makes far more sense to just leave it on all the time.
Here's an example, BTW, of a before & after shot. The before is with no hood. The after is with my hand substituting for a hood, since I didn't own a hood for the lens I was using (BTW,you do realize you can't use any old hood on any old lens - the hood has to be well matched to the lens or it might not provide any benefit). The main advantage of a hood over your hand is that it's more foolproof - half my attempts to use my hand resulted in a finger in the shot.
No hood:
With hand substituting for hood: