An update: I figured out what it was, and it was an issue with LaunchBox, although I have no idea why it came up and it might be my fault, although I have no idea how to figure out why.
I started by wiping my RetroArch install - no dice, didn't fix it.
I then tried to replicate the shortcut via commandline, and that didn't fix it either - but IT DID show that launching with the core option seemed to work just fine. So Launchbox was somehow launching with the wrong core option.
I dug into the .XML file for the emulators config and realized that I didn't have an emulator platform for Nintendo Game Boy, only Nintendo Game Boy Color. I have no idea how this happened, as I definitely successfully played some Gameboy games a while ago. Essentially when launching the game it wasn't specifying the core since there was no platform, but it didn't error or anything either; I'm actually surprised it was even trying to launch RetroArch at all, given the lack of a platform, but it was, and so that was causing the weird behavior and logged errors on RetroArch.
I dunno what caused it? Maybe some weird bug, or maybe I accidentally did it myself somehow, I think either are possible. In any event if someone finds this again later, here's what happened.