There's an easy fix for this
Retro808 is correct but a little confusing here's the easier process
Select Kega as your emulator
Got to the "Tools" menu in Launchbox
Select "Manage Emulators"
Edit Kega
Got to the "Associated Platforms" tab
And here you'll see your fullscreen problem clearly
Just delete the -fullscreen parameter for each console and Kega will not go fullscreen
I'm not a big fan of Retroarch in Launchbox; and this worked well for me