Found what was causing it. I left Any Key on, and didn't do anything with DetectJoysticks=0, however detectjoysticks=0 would have probably also solved it.
What I found, was that when I unmarked "Any Key", and hit "Pick", it would immediately say "J3UP". I was like, thats wierd, I don't have 3 joysticks and what was keeping it pressed up.
I unplugged all the USB devices attached, and it was still occuring.
I finally went to add/remove programs and found I was using a very old version of VJoy 1.2 which allows the keyboard to act like a joystick.
After uninstalling that, and rebooting (to clear out the VJoy joysticks you can find by going to joy.cpl, joystick control panel) the issue stopped occurring.
So it appears a bug in VJoy caused J3UP to be perpetually pressed, causing immediate end of voice feedback.