Add one of the AHK scripts below to the "Running Auto Hot Key" tab in your Project64 emulator setup in Launchbox.
This one is pretty standard and used most often.
; This section closes Project64 when pressing Escape
$Esc::
{
Process, Close, {{{StartupEXE}}}
}
This second version is a safer way to close an emulator. The first version force closes. It is best for emulators that have to close properly to ensure save states get saved.
$Esc::WinClose, ahk_exe Project64.exe