I don't know about modifying config files, but Retroarch hotkeys don't remap when switching controllers indeed, so here's what I've done:
Emulate all my controllers as xbox360 controllers and remap button appropriately using reWASD (I used Ds4windows but it started to be too limited at some point). This way controller buttons in Retroarch will remain consistent across all my controllers. You can also autodetect programs running with reWASD if for example while CEMU is running, you want to switch to a Dualshock 4 profile automatically if you have a gyro capable controller and want to use motion controls.
On top of that it wiil also allow you to navigate through BigBox with all controllers consistently (as long as you have enough buttons on the controller, and you can also use an autodetect remap with reWASD if needes), this was also something important to me.
Here's my controller list and all work perfectly like this in any emulator (Retoarch hotkeys are consistent) and BigBox:
2 DualSense BT
1 8bitdo M30 BT
1 8bitdo Arcade Stick BT
1 Exlene GameCube BT
1 Brawler 64 BT
I also have a Sinden Light Gun where I could also map Retoarch hotkeys to keyboard inputs also mapped on the gun, as Retroarch allows for a controller and a keyboard input mapping for hotkeys. I also managed to find a way to achieve some basic Bigbox navigation
The only controller I had to setup differently is my Dualshock 3 where I had to use the HidHideMini and Bluetooth drivers to be able to use pressure sensitive face buttons in PCSX2 and RPCS3 as it is the only way. I am not able to use BigBox with this one