hi i dont not if this is still relevant to you i had the same issue with LaunchBox/BigBox on an arcade machine. i want to us an Xbox controller for PSP, PS2, N64 and GameCube and as 3rd and 4th players on MAME games. when setting the emulators everything was great, but on LaunchBox when i press A and B together the game exits regardless of the emulator which drove me mad. and disabling the option "use all attached controllers" from LaunchBox was still exiting the games. i FIXED it by disabling the option "use all attached controllers" from LaunchBox and BigBox separately as BigBox does not sync with LaunchBox on this option.
and i enabled only the 1st player arcade controller to exit using Automation HotKey.