For N64 you are getting a black screen probably because you are upscaling the resolution and that messes with the shader. I would just suggest setting the resolution in the core settings to your monitors native resolution and be done with it. I think it looks about as good as you are going to get for an N64 game.
GLupeN64 is no longer a thing using that name, it is now back under the Mupen64plus core and the old Mupen64 core is under the ParaLLEi core. Use the Mupen64plus core now.
For other systems I use Retroarch where possible but for Dreamcast I use Demul, and PSP I use the stand alone build of PPSSPP. The stand alone emulators for these systems are better than the Retroarch cores.
Mame is a toss up between the RA core and stand alone when it comes to the latest versions. They are both very close in terms of quality with very minor differences separating them and neither has a definitive edge over the other. However if you are using an older version of a Mame core then the stand alone is a much better choice over the Retroarch core.