I am having a similar issue. I just setup my videos folders to use with BigBox and everything runs fine for a few minutes, but as I cycle through more games BigBox becomes increasingly slow and unusable. When I check the memory and CPU, BigBox is using upwards of 4-8gb of ram (I have 16gb total). If I close out of BigBox and reopen, everything is fine again but eventually devolves to the same issue. So I'm assuming it's not clearing the cache of images or videos over time.
I am also using VLC for the video playback, I have not tried using WMP.