For future me. In some Amiga multiplayer games such as super off-road 4x4. Port 1 needs to be mapped to Port 2. Port 2 needs to be mapped to Port 3. This is done via in game controls.
Turns out this issue was related to sensitivity of the Xbox elite series 2 analog stick.
Anything below 0.3 in input setting for analog stick input sensitivity within Retroarch the analog stick won’t register. N64 core needs around 0.5 to register and 0.8 to be acceptable for play.
I never thought this would be an issue.
This creates another issue because the Amiga core hates anything above 0.2 and like hotkeys you can’t create overrides or remap for seperate cores for these settings in Retroarch. The joystick is a good alternative for amiga core.
Writing this for future me as I’m bound to forget.
Elmo80's post in Retroarch - player 1 also controlling player 2 at the same time. was marked as the answer