MESS / Mame has controller issues with its 5200 emulation, for example, Pitfall you cannot start and Moon Patrol you cannot jump. For whatever reason these buttons just do not register with these games running.
I'd recommend using Kat5200 for that system, it's relatively easy to setup and works great even though the GUI for it is kinda oldschool and ugly, but you only see that when setting it up.