I've always considered that there is possibly a negative optical effect of using very oversized filters on a smaller lens - but I'll admit I've never done any testing to back up my theories.
However, if you envision your large filter mounted in a large step ring, a lot of the glass is sitting right in front of shiny black metal (most step rings are shiny anodized black). Some light must penetrate the filter and reflect off the black metal and bounce back towards the filter ... and possibly reflect back towards the lens.
Do you really end up with a ghost image of your step ring? Surely a good hood will help, but not on an ultrawide lens, of course.
Again, this effect may be entirely imaginary, or theoretical, and I've never tried it myself, as I usually carry filters close to the size of the lenses I have. But belief in the theoretical or imaginary has never stopped photographers worrying about such things .... especially when filters are involved.