I think you need to understand the differences in how a CRT handles these images compared to how a modern fixed pixel display handles them.
There is nothing in Mame you can do in terms of resolution to help with this, the best you can do is to use HLSL shaders to try and replicate the CRT as closely as possible. A modern display will enlarge the image which was typically 320 pixels by 240 pixels but of course it could vary. A CRT handled this quite easily because of how a CRT works but a fixed pixel display which is much higher resolution has to scale these images up quite a lot, 1080p, 1440p or 2560p. Open up a 320x240 jpg image and enlarge to fit on your display and see what happens, this is similar to how an emulator scales the art in these old games.
Here is a link to my HLSL CRT shader settings I use in Mame:
Also here is a video that hopefully can do some explaining of this kind of stuff:
Even though it is more a discussion of aspect ratio it covers much of the same issues happening here for you.
Also, here is a screenshot of Street Fighter using my HLSL settings.