Someone else got it working with XBMC Advanced Launcher, and I just copied their method to get it working on LaunchBox.
Make a .bat file with the following contents:
cd "C:\Emulators\N64\Project64 1.6"
SET STRING=%1
SET STRING=%STRING:"=%
start Project64.exe %STRING%
(Of course, you should change "C:\Emulators\N64\Project64 1.6" to whatever path your Project 64 install is.)
Then, when you add the emulator to LaunchBox, you point to the .bat file instead of the emulator executable.
Source: http://www.gwenael.org/forum/viewtopic.php?id=56
P.S.: The installer for newer versions of Project64 is full of malware, and even though you can opt-out the unwanted toolbars, some malware will still be installed (like iWebar, Shopper Pro, and YouTube Accelerator), so I recommend you stick with either version 1.6 or 1.7, and download it from somewhere other than the official site, just to be safe. I use 1.6 because 1.7 refuses to work on my machine, no matter what I do.