Question about metadata in database versus xml files

[Rinusch]

Hi,

This is not so much a 'troubleshooting' topic but I don't see any forum that better matches this.

I notice that some media files for arcade games have a 36-character long ID appended to the name of the game, e.g. 'Tetris.0fb24d18-34d5-4b9b-98f6-55f1a33e7fcf-01.png'.
I understand the reason for this, which is being able to differentiate  between different versions of games having the same 'base name'.
In order to link the right media files to the right games, this ID has to be translated 'back' to the LauchBox DB ID for the game. Because I couldn't find this ID in the SQLite database structure, I searched in the XML files and found it in the 'arcade.xml' file. This is where the translation is made from this ID back to the LaunchBox DB ID.
However, I also notice that this arcade.xml file (still) holds the metadata for the game, where the SQLite DB also holds metadata for the same game. The data in the xml file seems to be out of date.

In case of this particular game, the community star rating count differs between the SQLite DB and the XML file for instance (resp. 174 and 164). Launchbox apparently takes the (possibly outdated) values from the XML file (164) when displaying detailed game information.
Are there any short-term plans to move away from using this XML file and to solely rely on the SQLite DB (and incorporate these ID's as additional fields there)?

Regards!

Edited Wednesday at 11:26 AM by Rinusch

[C-Beats]

1 hour ago, Rinusch said:

Hi,

This is not so much a 'troubleshooting' topic but I don't see any forum that better matches this.

I notice that some media files for arcade games have a 36-character long ID appended to the name of the game, e.g. 'Tetris.0fb24d18-34d5-4b9b-98f6-55f1a33e7fcf-01.png'.
I understand the reason for this, which is being able to differentiate  between different versions of games having the same 'base name'.
In order to link the right media files to the right games, this ID has to be translated 'back' to the LauchBox DB ID for the game. Because I couldn't find this ID in the SQLite database structure, I searched in the XML files and found it in the 'arcade.xml' file. This is where the translation is made from this ID back to the LaunchBox DB ID.
However, I also notice that this arcade.xml file (still) holds the metadata for the game, where the SQLite DB also holds metadata for the same game. The data in the xml file seems to be out of date.

In case of this particular game, the community star rating count differs between the SQLite DB and the XML file for instance (resp. 174 and 164). Launchbox apparently takes the (possibly outdated) values from the XML file (164) when displaying detailed game information.
Are there any short-term plans to move away from using this XML file and to solely rely on the SQLite DB (and incorporate these ID's as additional fields there)?

Regards!

The ID you're seeing in the image file name is an unique ID unique to that record and your install which is why it isn't in the Sqlite files. The XML files are your local library metadata, the sqlite is the information on our games database website. The two are intentionally separate so that you can alter your data the way you want on your build. I for example do not enjoy how we do genre tagging in our DB and so have manually corrected it on every game in my collection. I couldn't do that if I had to use values in the Sqlite file. Eventually we may move your local library data to Sqlite but it will still have that degree of separation and not all be in the same file.

For your star rating disparity you can download the newest scores via the tools menu inside of LB. You can also always update other metadata using the Download Metadata and Media wizard.

[Rinusch]

Thanks for your quick response, it is appreciated!

Ok, so the concept of this sounds good: being able to 'override' the default data with your own specific data.
However, why do I have so much data in all of these XML files for games that I never intentionally edited data for or even touched upon? And also: why would you want to store not only the fields you've changed in the xml but seemingly all fields, including a volatile data element such as the star rating count recorded in the XML? Overriding this value wouldn't make sense, so why is it recorded there and why does LB display that value instead of the most actual value in the SQLite DB?

Is the XML data (which I assume to be placed in memory at load time) interpreted every time game details need to be presented to the user, e.g. when selecting a game or when editing its metadata? If so, this sounds like a possibly noticeable performance hit, especially when there's XML data present for so many games and not only the ones I have previously manually overriden data for.

Don't get me wrong, I'm just trying to figure out how this works and maybe hope to help somewhat in addressing potential performance issues. 😄

Regards!

[C-Beats]

8 minutes ago, Rinusch said:

Thanks for your quick response, it is appreciated!

Ok, so the concept of this sounds good: being able to 'override' the default data with your own specific data.
However, why do I have so much data in all of these XML files for games that I never intentionally edited data for or even touched upon? And also: why would you want to store not only the fields you've changed in the xml but seemingly all fields, including a volatile data element such as the star rating count recorded in the XML? Overriding this value wouldn't make sense, so why is it recorded there and why does LB display that value instead of the most actual value in the SQLite DB?

Is the XML data (which I assume to be placed in memory at load time) interpreted every time game details need to be presented to the user, e.g. when selecting a game or when editing its metadata? If so, this sounds like a possibly noticeable performance hit, especially when there's XML data present for so many games and not only the ones I have previously manually overriden data for.

Don't get me wrong, I'm just trying to figure out how this works and maybe hope to help somewhat in addressing potential performance issues. 😄

Regards!

Some of these have been the design of the app well before I started but I'll do my best to explain the reasoning as best as I understand. First off being the separation of files. When LB was first built we didn't HAVE the database and so you HAD to store information locally in the data files because when you scraped you were hitting a third parties API and not this file. Updating to "new data" from any data source is by design a manual process as well because people get REALLY irritable when the software starts changing their data all willy nilly. It's because of those design decisions each field doesn't have any metadata tied to it saying, "user manually touched this, please don't update". We also don't have any automated scraping processes that would update the data for you even if we did have that flag.

The data from the XMLs is shunted into memory on load. Obviously not in raw text, but in "models" we use internally in code. This means that we are only susceptible to drive read failures negatively impacting the app once, and not every time we want to do any load process. It's also WAY faster to read from memory than from disk, the difference becomes greater on spin-drives and network locations. Having all the information in one file also means that we only have to load from one source, and not two (which we'd then have to merge) which is far faster and leaves only one point of potential failure instead of two. There is also messiness regarding game entries in your library NOT in our database, or entries that started off NOT in it and then since were added and connected, the fact that metadata came from other sources than just this file (like MAME ini files, storefront APIs, etc), and more...

There are definitely things we've identified as things we'd do differently in the future if we had the opportunity, as well as some nice have things that would make certain tasks easier for both user and dev. As time allows we push the app towards those goals, but some of those legacy decisions are rooted DEEP in this code base and so migrating to our newer desires can often time being a MUCH larger and complicated task than you may assume and so get pushed off for other area's of the app we can improve in a more timely manner (that also typically have far larger impact on the overall user experience).

Quite a large number of the performance bottlenecks have been identified and I've offered several solutions to combat, but as the only developer actively working on the PC version of LB there is only so much time in the day and so we have to really triage where and what I do in the codebase for maximum impact to everyone.