I definitely prefer to shoot with lenses that introduce distortions of one sort or another, and I do think that it adds character. My most used lens by far is the Super Takumar 20mm, which has a complex "moustache" distortion that I never correct. And it could be argued that all my Takumars have a "distorted" colour rendering compared to the computer-designed neutrality of modern lenses, but that's exactly why I choose to use them.
I quite often defish my fisheye shots though, but I don't consider that to be removing distortion. The defishing plug-in that I use (PT Lens) merely stretches out the corners of a fisheye shot so that it looks more like the characteristic perspective distortion of a standard ultra-wide. There's still distortion in the photograph, but a bit of defishing lets me choose how much. Here's one from a few days ago with the Takumar 17mm fisheye. It has been defished, and was intended that way all along, but there's still ultra-wide distortion as an essential part of the composition.