Let me follow up to this... After additional troubleshooting, I've realized that exiting out of MAME64 it doesn't give focus back to Bigbox so the app doesn't start running again. (Even though it's full screen)
Mouse click brings it to focus and running as expected. An alt-tab does the same thing. Exiting out of a Retroarch game works as expected, so I guess it's a quirk of MAME64...
Suggestions?