That is a issue with using MAME inside of Retroarch. F4 is the default load state key for Retroarch, but with the MAME core loaded that takes the input of F4 instead, which shows the palette screen, just as it would on standalone MAME if you pressed F4 during a game. MAME has its own save and load state buttons.