I put the raster.ini in the \Mame\ini\presets folder. There was also another file in the download called vector.ini. I just put them both in the \Mame\ini\presets folder. I backed up the old ones. The HLSL effect doesn't show up when running in BigBox, it only works when I manually run MAME and select the game that I want. It's a really cool effect, I hope there's a way to get it working in BigBox.
Btw, are there any other resources that have custom HLSL MAME shaders? I’d like to try some out and see what I prefer.