I've finally found the core of the problem- it lies with the DNS.
This domain was registered less than 72 hours ago, which is the typical time for a domain to stabilize. To my surprise, I could access the domain after 5 minutes of registering it...so I jumped right in with the site.
So, at about 6 PM today, the delays should all be gone
They appear differently on different internet connections- some people won't see any delay at all, while some will have to wait several seconds on certain pages...but it'll be over soon
I've also done other optimizations that should make everything else load as fast as possible.