I have no experience with arcade CRT monitors but according to this: https://www.arcade-museum.com/monitor.html the standard res is 336x240 not 320x240. This small difference can can make all kinds of sync errors. I guess the reason why 640x480i works fine is that it's overdefined, and gets downscaled at one point, so it looks better on the standard resolution.
But then again, Im just guessing here.