bug Custom fields duplicating, games disappearing, videos disappearing

[xevidroid]

Something very funky has been happening to my Launchbox. I've provided a screenshot of one of the occasions where custom fields are being duplicate, this seems to affect only some games and only some fields, and to get worse over time. I spent 3 hours setting the game video for these games, but the video source keeps getting reset, despite the games being present in the respective folders. I've also lost my last few added games, but I really can't say for sure if something else is missing. This has started happening today, after turned on my PC. Last time I had used launchbox it was on my other PC, and the two folders are synced through Google Drive. The problem can't be Drive though, because despite the xml files sometimes conflicting, the issue is in one xml file only, the one in use. For example if I open the custom field entry for 60 seconds, I can see (at the bottom of the Windows.xml file) 8 different entries of the game's id + a repeated field.

 

This is very frustrating and potentially distructive to my setup which I put hundreds of hours into. I really can't keep using the program until it gets fixed because I risk destroying my collection and further corrupting my xmls...and I really don't have any way to check if or what is getting screwed up or deleted, which really drives me insane )= has this ever been reported before? I can share my ENTIRE setup with the devs on request, please contact me if needed and I'll share the whole installation folder.

[xevidroid]

I'm currently investigating the issue. I was lucky enough to find out that my copy of the folder on the other PC was still unaffected by the bug. I made a backup and then moved a copy to the PC where I initially found the issue. The first time I open Launchbox, everything's fine and the games only have 1 entry. After closing and reopening Launchbox, the data had been duplicated. Closing and reopening again didn't duplicate it a second time though. I'm trying different combinations of opening and closing to see when the bug triggers. I'm also trying to find out if the videos disappearing is part of the same bug, or is something completely unrelated. I'm also going to find out if the issue affects the laptop too, or only happens if the computer that is opening the program is the desktop.

 

Findings:

If I set a video path for some games, it gets immediately removed after one restart of the program, EVEN when the duplication bug doesn't happen. Some other games retain the videos, even after restarting when the duplication does happen.

Only ONE platform seems to be affected by the duplication glitch (currently it's windows, I've yet to see it happen for other xmls).

The video deletion bug seems to happen at random just like the duplication one. Sometimes you reload and the video is there, sometimes it is not.

The "duplication" happens with no discernible rules to the amount of times the items are duplicated. Sometimes it stays the same, sometimes it jumps from 2 to 7, or 8, sometimes it goes down to 2 or 3. I've never seen it go back to 1.

Some games (even NOT in the Windows platform) lose the video instantly, after one restart, and never get it back. Some other games I add the video and when I restart, sometimes it's there, sometimes it's not. For games that never get it back, the video is not found in the xml. Some games that have it disappearing and coming back, retain it in the xml file. 

The duplication of the custom fields CAN be seen in the xmls. It changes every time it changes in the program, differently from the videos.

 

In short, it's a big big mess. As it stands the program is unusable, because any change I make is in constant danger of being ruined or deleted. I can't add videos because they vanish, I can't add games because I've seen some vanish too, I can't edit information because I simply can't trust the xmls and I don't know if I'll have to roll back to my old backup. I don't know if I'm alone in this and if the program uses a separate database from the xmls and that's what got corrupted for me and simply resetting that will fix everything..I simply don't know. Please get in touch and let's solve this together, because it's really a dealbreaker and if this can happen to random customers, it's a big, big issue. If anything, we'll find out if Google Drive was the initial cause of the issue (maybe some file was lost in synchronization and now the whole installation is busted), and installation inside cloud folders could get discouraged from now on.

 

[DOS76]

@Jason Carr

[xevidroid]

I've tested the laptop. The duplication didn't seem to happen at first. Then I tried adding a video to a game that DIDN'T show the duplicating behaviour on the desktop, but DID delete the video every single time I tested. The moment I tried and restarted, the video disappeared and the other games started duplicating. I don't know if this means that the duplication is triggered by adding the videos, but what I do know is that when I noticed it had started happening, I was in the middle of video adding spree, and I had added something like 50 videos in a short amount of time...so they may be related. It's VERY weird. I can't think of any other test to do myself, I'll wait for an answer here. On request I can share my entire folder, in the version with or without the bug already in place. I also have a backup from a few months ago, I don't know if it can help. 

 

One thing I could try is installing the program in an empty folder (maybe even an older version), then moving the data (xmls, images and videos). Still, I'll wait for a response.

[xevidroid]

One last test: I went back to my backup from version 9.8, the only backup I still have around to my knowledge. The duplication bug is NOT there, I'm 99% on this because I tried many times and the information hasn't ever duplicated once. I also tested the videos, and adding a video to a game that did lose it on 9.10, resulted in it losing the video on 9.8 too. The bug seems to be present, but unrelated to the duplication one. I also tried to go on and update the 9.8 folder directly to 9.10 (didn't go through 9.9 like I did originally), but that by itself didn't trigger the bug, at least for now, so the answer might reside in the difference between my May 2019 and August 2019 folders. The video kept disappearing after the update to 9.10, at first I thought it was because I was putting videos in the wrong platform folder under Videos (not that it should matter too much, I thought), but it turns out that even by changing folder the video still disappears. It also seems to happen to specific platforms: Android games always lose their videos, but weren't affected at all by the duplication glitch. Windows games never lose their videos but get duplicated. Android games that had videos PRIOR to all of this starting to happen, seem unaffected by the video disappearing bug, as is any other game that already had a video. Might this be related to an encoding error in these two specific platform xml files? The XML is definitely valid, but it may be a Launchbox internal error and not a syntax one.

Edited August 8, 2019 by xevidroid

[xevidroid]

Today I was quickly browsing my Launchbox collection to check if anything else was out of place, and I noticed something alarming. SOME (but not all) of the videos that had been disappearing from the game details, had been physically deleted on my disk. This means that some of the stuff I put into Launchbox, somewhere, could at any time completely vanish and for those videos I didn't even have a backup. This issues NEED to be investigated and corrected, just because it's happening to one person (that we know of), it doesn't mean it couldn't start to affect others. I've seen people with libraries of thousand of games, people who have put even more time than I have into completing their setup..These bugs are putting hundreds of hours of work into danger! Please get in touch so that I can send you all the necessary information!

@Jason Carr @DOS76

[neil9000]

Have you ran a virus scan at all? Files just disapearing from your drive sounds suspiciously like a virus, and as we have no one else report anything like this before, it does seem to be very specific to your system.

[DOS76]

have you error checked your drive. Open file explorer got to This PC right click the drive in question then go to properties. Go to tools and then run the error check and even if Windows tells you that you don't need to run it run it anyway. When if finishes it will give you an option to see details (if it finds errors I believe) click that and it will open event viewer there it will tell you if the error check fixed everything or if you need to run a chkdsk /f on the drive or not

[Jason Carr]

@xevidroid LaunchBox does not and has never randomly deleted video files (or image files, or anything else, for that matter). That has literally never once been an issue in the past and it never will be; it doesn't happen. What this means is that there's something else on your system that is causing serious issues like this. It could be anything from drive failure, to a corrupted file system, to a virus or malware, or even something strange like an app conflict. I'm sorry you're running into these issues, but stuff like this is not caused by LaunchBox. Let us know if there's anything further we can do to help identify the root problem here.

[xevidroid]

@DOS76 Yes I error checked my drive and that's not the issue.

@Jason Carr Okay, the videos disappearing might be caused by anything else (Windows? I've seen it happen before), but I've reported a big number of anomalies that are clearly related to the inner workings of Launchbox and could not be caused by something else in my system. For example, every time I open or close the program, my custom fields get duplicated a random number of times, like in the picture I posted. If I stop there and read the XML file, the fields have been duplicated on a "database" level too.  It's not a read error or a virus, it's literally the program saving a wrong value on its database. The same goes for the entry for videos disappearing, the URL gets deleted from the XML even though the video is still physically present in the folder. As I said, I can send you the zipped folder with and/or without the bug and you can check for yourself that it's not something in my system in particular. My best guess would be read and write errors when parsing or updating the XMLs, due to something unusual in my particular library, maybe some unusually heavy media resources, maybe some funky encoding in some text somewhere. The program clearly struggles to load my 650 games, as I noticed that sometimes when I open it, it says I have 649 and some game is not shown, but it's back there if I restart. I'm not trying to insinuate it's bad programming or someone's fault, bugs happen and I love Launchbox, I wouldn't have paid for the full version or posted here if I didn't care

 

Just to be clear, my setup doesn't include any ROM nor any personal information, just images, videos and a list of games I have played or I legally own. I think it wouldn't be a problem to share it.

Edited August 10, 2019 by xevidroid

[Jason Carr]

@xevidroid It is hard to say at this point where the problems lie, but I will say that you're running into a bunch of issues that I haven't heard of anywhere else, which still makes me think they're somehow related to system instability. I have well over 10 years of programming and troubleshooting expertise, and I know better than to try and track down bugs on a system that is just obviously unstable. You have to track down one bug at a time, or else you'll just end up spinning your wheels. Even if there are legitimate bugs in LaunchBox related to your setup, I can't really do much to track them down or trust any results until you're working with a stable system. My apologies, but it really would just be wasting my time if I dove in deep when there's obviously more serious issues at play.

[Jason Carr]

@xevidroid One more thought that @Retro808 brought to my attention. Google Drive has the potential to mess everything up, if it's syncing files incorrectly. It's been years since I tested Google Drive, but back when I did it had some major, major problems with things not syncing properly. If I had to guess, it likely has to do with Google Drive. It's worth moving LaunchBox out of Google Drive to see if you are running into the same issues.

[xevidroid]

@Jason Carr The thing is, I seriously doubt my system is the culprit. If I move the folder to another PC (manually, through a USB stick) I see the same bug. I see the same problem whether I load from an old HDD or a fast SSD. I don't have any problems with any other software installed, everything runs smooth..So I either have a very specific Launchbox killer virus, or it's something else..

 

Google Drive COULD have messed things up, it's totally possible. I don't know what precisely could have gone wrong, but I do see how syncing 11k files through a garbage app like "Backup and Sync" could cause errors. The problem is though, I'm not using Drive anymore, the folder is on my desktop and I'm only moving it through USB to test things out. Basically Google Drive could have messed up some file or dependency and started the problem, but it's not the thing that actively causes issues at runtime. My programmer instinct tells me there is a deeper problem with how the read/write of the XML is carried out, but as the source is not on Github I can't check for myself. 

 

As I said multiple times, if you don't trust my system, or you don't trust my word or whatever, the only solution I see is I send you my setup and you open it in a sandboxed VM or whatever and check it out for yourself. Either this, or at the very least I'd like some suggestions on how to solve the issues once and for all, because after 60$ and hundreds of hours put into designing my own box art covers for each of those 650 games I really don't want to give up Launchbox like this, and starting over would be insane, especially if I know the bug could still happen at random in the future.

[Retro808]

Have you tried a clean install at all? If your Launchbox is corrupted somehow and you are just transferring the same files to a different drive or pc it is still corrupted files. Sure, I am looking at it in simple terms of troubleshooting, but without knowing if you have tried this it is a missing piece. I would test a fresh install, import only a few games and test making changes to those games to see if it still occurs and I would try on a different drive and folder location.

I am not saying it is in no way possible a bug (anything is possible), but the probabilities of your specific system finding a bug that no one else has experienced in the years Launchbox has been in action is pretty astronomical. 

[xevidroid]

Okay so, I had already tried my backup from 9.8 and it didn't have the duplication bug, but it did have the video deletion bug. I tried a fresh install of the 9.10 version and then adding the same games that were having problems in the same way, but the bug didn't happen. I then tried adding my whole data folder to the new install and the bug wasn't there, at least that I could see. I added the whole images folder and the bug wasn't there. Same goes for my videos and music folders. Basically there has to be something that has gone wrong in my current folder, and I don't know what. I could simply do a fresh install etc, but

A) would this work? @Jason Carr which folders do I need to move to the new install to have a perfect 1:1 copy of the old database + media? 

B) This doesn't solve the issues with my current bug, because if it happens again, we're back to square one. Wouldn't it be better to solve the problem once and for all by finding the culprit? I mean, I've had my library basically vandalized, videos disappearing, changes not saving, games disappearing, information duplicating..all in one go. This is pretty destructive, it really should be investigated..

[neil9000]

4 minutes ago, xevidroid said:

B) This doesn't solve the issues with my current bug, because if it happens again, we're back to square one. Wouldn't it be better to solve the problem once and for all by finding the culprit? I mean, I've had my library basically vandalized, videos disappearing, changes not saving, games disappearing, information duplicating..all in one go. This is pretty destructive, it really should be investigated..

Well the thing is that it doesnt sound like a bug, as you said yourself a fresh install doesnt exhibit this "bug" so it still looks like something else on you system specifically that was causing the issue in the first place. 

[Retro808]

1 hour ago, xevidroid said:

 B) ...Wouldn't it be better to solve the problem once and for all by finding the culprit?... 

Sounds to me like your old launchbox build got corrupted and is the culprit.  

New install plus fresh data = no issue.

New install plus copying over old data = issue. 

I would continue to test the fresh install adding games and editing data to see if it stays issue free or does not. 

[xevidroid]

23 hours ago, Retro808 said:

Sounds to me like your old launchbox build got corrupted and is the culprit.  

New install plus fresh data = no issue.

New install plus copying over old data = issue. 

I would continue to test the fresh install adding games and editing data to see if it stays issue free or does not. 

@Retro808 I'll try that and report back. 

[xevidroid]

@Retro808 @Jason Carr Ok, after a few weeks of testing I can say with 100% certainty that I've fixed the problem, and I'm pretty sure I know what it was. As some of you have suggested, the culprit was Google Drive: I originally dismissed that option, because I had a pretty good idea of what the internal file structure of my data folder was and it hadn't changed much for months, as far as I could see; I always had those xml files with the [conflict] tag, but since they hadn't been causing any issues for months or even years, I figured Launchbox was simply ignoring them (because they had the wrong name, why read "windows[conflict].xml" when "windows.xml" is already there as expected). As it turns out (and I don't know if this is something new Jason introduced with 9.10), Launchbox is now VERY confused by the presence of those files. First, the problems were only there for platforms with conflict files in the data folder. As for the duplication issue, it appears LB was reading the custom fields from the xml array in the conflict files AS WELL as in the regular file, but when closing the program, it was probably writing back the fields to one file only, thus completely messing it up and multiplying the lines. I guess the read happened from one file or the other at random, which caused the shifting number of duplications. As for the videos disappearing, I guess that when adding videos LB was committing the changes to the wrong file, or not committing them at all due to the conflict being there. These two are only my guesses based on outside observation, what's 100% sure is that by going back to an older backup of the data files without the duplication, and then deleting the conflict files, the issue went completely away and hasn't returned.

So, my issue has been fixed. What I suggest is either discourage using Google Drive as the main directory for Launchbox (at this moment in time, the Google Drive symbol appears on the description page for LB on the website), or if possible writing the xml parsing code in a way that simply ignores duplicated platform files, or specifically platform files with "[conflict]" in the name, as at this moment in time LB seems to basically merge the two data structures together at launch and then go crazy when figuring out what or where to commit changes. I hope this proves useful and avoids the same happens to other users in the future (as it was pretty dishearting for me).

Edited September 4, 2019 by xevidroid
spelling