Thanks @SentaiBrad, I came up with a solution though. Found two button numbers common to all three of the pads I use, my iBuffalo SNES pads, Gamecube, and 360 pad. Now, if I have either the SNES or 360 pads connected, holding Start + A exits. The corresponding buttons on the GC pad is holding Z + Y. Hopefully I don't encounter any games that would use those button combos!
I'll log a ticket for per controller configs though, that would be really beneficial for users I think. I know some people like to use original pads for systems, as long as the pad is connected before launching Big Box it would be fine, but hotswapping would be even sweeter ?