Yeah, I hear what you are saying and I don't experience the issues you describe. I can change platforms which contain several hundred games (NES, Genesis, SNES) and it will scroll smoothly.
I use Launchbox every day, it is always loaded even if I am not using it and this is never a problem at all. The worst that happens is there is the occasional box art that needs to cache in for a micro second but once it does it's fine.
Even on my low end HTPC it runs fine, though the library size on that system is much more streamlined.