Ok I've been messing around with this. The first problem with doing this on Retropie is that if you use the internal MAME artwork system then your retroarch shader applies to everything - bezel, backdrop, and screen. And even if you were cool with that you're only going to get the best results with lr-mame-2015/2016, and I'm using a pi3 and those cores just tax the system too much, especially when you add artwork to load in.
But the hack i've seen other people do is you load the bezel through retroarch and then the overlay/backdrop in MAME. Then your shader won't affect the bezel.
Let's start with the space invaders backdrop:
The .lay system in lr-mame-2008 just behaves very differently so I had to change the PAR to get it to look right, but it didn't look much better than lr-mame-2003 so I went 2003 and the old .art system for the backdrop.
The only downside is the default MAME red/green overlay is visible (you can see the rectangular outlines) but I don't mind it. Oh and the backdrop is a little pixelated but again we have to bump up to mame 2015 at least to fix that - you have to enable "alternate render method" in the core options and it kills your FPS.
For Asteroids Deluxe on a Pi3 the best core I've found is AdvMame. You can really dial in your vector settings and get things looking great. But then you cannot use RetroArch at all so you need to load the bezel AND backdrop the .art file way. So I made a whole new bezel for it while I was at it.
Anyway - learning experience for sure. I'll organize all these files and post them to the retropie forum and link them here or something. (The backdrop on asteroids looks brighter in the picture than it is.)