So I did a little googling, but still can't find a solution to my problem.
If I fullscreen then exit via ALT+4 or Task Manager it works and I am back to Big Box, BUT this is where my issue lays.
When I start ANOTHER game and exit with controller automation it returns back to the windowed mode.