I believe it does matter, because physics.
Photographic techniques convert light into a recording medium, and every recording medium I know of has a "saturation" of too much light where it can't record any more.
So, my basic reasoning is that _if_ you are limited in your imaging by noise at base ISO (e.g., you have plenty of light to work with or can increase it in a studio setting or something, and are viewing at an enlargement factor that makes noise problematic for you) then the increased sensor area can allow for more electrons to be captured providing a better overall SNR.
I think the reasoning that gets confused a lot is "limited in imaging by noise at base ISO" -- and the expectation that somehow lenses can be perfectly matched across formats. So I think what is often compared is something like, an 85/1.4 35mm vs. a similar (in terms of min DoF) 120/2 (made up numbers, should be about right) on a larger format -- will have same DoF and total light gathering capability wide open though the 120/2 image will be dimmer of course -- you'd have to apply one stop of gain in post and exactly give up the one stop of "better noise performance" unless you also changed the shutter speed to be longer. Now, with that shutter speed the smaller format would "blow out" and lose information while the larger format, spreading that light out over more area, can capture all of it. However, modern techniques such as Pixel Shift and HDR or multi-frame image stacking etc. can all be used to capture more data over more time with the same imaging system at the cost of needing more computation/time. With pixel-shift, you gain a stop of data at the cost of artifacts, which would be rendered as blur in the "normal" exposure of equal brightness and information (given same image rendering choices) you could do with a larger format. Or, if you want to change DoF, maybe there's a larger format lens that has a wider equiv. aperture and you could get more light per time captured keeping the shutter speed the same.
The basic principle boils down to: more area = more recording capacity per shot. But you don't need more recording capacity if you're not able to fill up what you have already! (I.e., the way you shoot involves shooting above base ISO.)
One of the reasons I believe film can be superior to digital is related to sensor size, but the other is related to film speed -- you can get some low speed film that has exceptionally low grain/noise (mmmm, velvia 50), which will saturate slower than digital sensors (base ~100 usually).
I think a lot of the perception comes from the early days of computer aided design, before the crazy types of and purity of glass, etc. In this context, where lens flaws also get enlarged, lower enlargement ratios would be far superior particularly on the wide angles where the light simply has to bend less for larger formats to get the same FoV. With the amazing modern giant hunks of glass we have, I think a most of the problems with smaller formats have been reduced significantly.
... Finally, I want to just say that understanding the technical aspects of it probably just make my life worse because it's easy to focus on and matters like 1% for creating better images at this point in my photography. If you're happy with the images you create in the form you choose to display them... what else really matters? It sounds like from your context paragraph, that your current systems are plenty for your choices. I found that 24x30 enlargements from APSC were a little iffy and I wanted to move to FF. Anything beyond that is fun for me but not what I consider "production" because I can't get modern glass for it (and the digital 645 systems don't represent what seems like a good tradeoff on my shooting which is largely non-studio based).
(My personal context: I started on 35mm film, went to APSC digital, then moved to FF on the K-1, added the pentax 6x7, and am considering getting into 4x5 or 8x10 but daaaang things get expensive and heavy.)