I did some more digging but in a different direction.
For me, the controls were good for some games automatically but for others not at all.
Inside MAME itself you could see that where I had my default for player 1 button 1 set to a key, there was Joy A behind it.
For other games it was the same key but with Joy Y behind it for some reason.
Searching for something similar brought me to a Reddit post telling to fix this you should turn off the core option named 'profile buttons according to game'.
I did that, cleaned up all my configs including the default, setup the general keys and presto, it works!
Hope this helps anyone coming across the same problem!