This also happens to me, as my collection is quite large as well.
What I do is have five separate LaunchBox's, one for Arcade, Computer, Console, Handheld and Pinball. Depending on which LaunchBox I use, the times vary from 3 seconds to around 6 seconds now. (before it was around 15 seconds)