Question about metadata in database versus xml files

[Rinusch]

Hi,

This is not so much a 'troubleshooting' topic but I don't see any forum that better matches this.

I notice that some media files for arcade games have a 36-character long ID appended to the name of the game, e.g. 'Tetris.0fb24d18-34d5-4b9b-98f6-55f1a33e7fcf-01.png'.
I understand the reason for this, which is being able to differentiate  between different versions of games having the same 'base name'.
In order to link the right media files to the right games, this ID has to be translated 'back' to the LauchBox DB ID for the game. Because I couldn't find this ID in the SQLite database structure, I searched in the XML files and found it in the 'arcade.xml' file. This is where the translation is made from this ID back to the LaunchBox DB ID.
However, I also notice that this arcade.xml file (still) holds the metadata for the game, where the SQLite DB also holds metadata for the same game. The data in the xml file seems to be out of date.

In case of this particular game, the community star rating count differs between the SQLite DB and the XML file for instance (resp. 174 and 164). Launchbox apparently takes the (possibly outdated) values from the XML file (164) when displaying detailed game information.
Are there any short-term plans to move away from using this XML file and to solely rely on the SQLite DB (and incorporate these ID's as additional fields there)?

Regards!

Edited June 4 by Rinusch

[C-Beats]

1 hour ago, Rinusch said:

Hi,

This is not so much a 'troubleshooting' topic but I don't see any forum that better matches this.

I notice that some media files for arcade games have a 36-character long ID appended to the name of the game, e.g. 'Tetris.0fb24d18-34d5-4b9b-98f6-55f1a33e7fcf-01.png'.
I understand the reason for this, which is being able to differentiate  between different versions of games having the same 'base name'.
In order to link the right media files to the right games, this ID has to be translated 'back' to the LauchBox DB ID for the game. Because I couldn't find this ID in the SQLite database structure, I searched in the XML files and found it in the 'arcade.xml' file. This is where the translation is made from this ID back to the LaunchBox DB ID.
However, I also notice that this arcade.xml file (still) holds the metadata for the game, where the SQLite DB also holds metadata for the same game. The data in the xml file seems to be out of date.

In case of this particular game, the community star rating count differs between the SQLite DB and the XML file for instance (resp. 174 and 164). Launchbox apparently takes the (possibly outdated) values from the XML file (164) when displaying detailed game information.
Are there any short-term plans to move away from using this XML file and to solely rely on the SQLite DB (and incorporate these ID's as additional fields there)?

Regards!

The ID you're seeing in the image file name is an unique ID unique to that record and your install which is why it isn't in the Sqlite files. The XML files are your local library metadata, the sqlite is the information on our games database website. The two are intentionally separate so that you can alter your data the way you want on your build. I for example do not enjoy how we do genre tagging in our DB and so have manually corrected it on every game in my collection. I couldn't do that if I had to use values in the Sqlite file. Eventually we may move your local library data to Sqlite but it will still have that degree of separation and not all be in the same file.

For your star rating disparity you can download the newest scores via the tools menu inside of LB. You can also always update other metadata using the Download Metadata and Media wizard.

[Rinusch]

Thanks for your quick response, it is appreciated!

Ok, so the concept of this sounds good: being able to 'override' the default data with your own specific data.
However, why do I have so much data in all of these XML files for games that I never intentionally edited data for or even touched upon? And also: why would you want to store not only the fields you've changed in the xml but seemingly all fields, including a volatile data element such as the star rating count recorded in the XML? Overriding this value wouldn't make sense, so why is it recorded there and why does LB display that value instead of the most actual value in the SQLite DB?

Is the XML data (which I assume to be placed in memory at load time) interpreted every time game details need to be presented to the user, e.g. when selecting a game or when editing its metadata? If so, this sounds like a possibly noticeable performance hit, especially when there's XML data present for so many games and not only the ones I have previously manually overriden data for.

Don't get me wrong, I'm just trying to figure out how this works and maybe hope to help somewhat in addressing potential performance issues. 😄

Regards!

[C-Beats]

8 minutes ago, Rinusch said:

Thanks for your quick response, it is appreciated!

Ok, so the concept of this sounds good: being able to 'override' the default data with your own specific data.
However, why do I have so much data in all of these XML files for games that I never intentionally edited data for or even touched upon? And also: why would you want to store not only the fields you've changed in the xml but seemingly all fields, including a volatile data element such as the star rating count recorded in the XML? Overriding this value wouldn't make sense, so why is it recorded there and why does LB display that value instead of the most actual value in the SQLite DB?

Is the XML data (which I assume to be placed in memory at load time) interpreted every time game details need to be presented to the user, e.g. when selecting a game or when editing its metadata? If so, this sounds like a possibly noticeable performance hit, especially when there's XML data present for so many games and not only the ones I have previously manually overriden data for.

Don't get me wrong, I'm just trying to figure out how this works and maybe hope to help somewhat in addressing potential performance issues. 😄

Regards!

Some of these have been the design of the app well before I started but I'll do my best to explain the reasoning as best as I understand. First off being the separation of files. When LB was first built we didn't HAVE the database and so you HAD to store information locally in the data files because when you scraped you were hitting a third parties API and not this file. Updating to "new data" from any data source is by design a manual process as well because people get REALLY irritable when the software starts changing their data all willy nilly. It's because of those design decisions each field doesn't have any metadata tied to it saying, "user manually touched this, please don't update". We also don't have any automated scraping processes that would update the data for you even if we did have that flag.

The data from the XMLs is shunted into memory on load. Obviously not in raw text, but in "models" we use internally in code. This means that we are only susceptible to drive read failures negatively impacting the app once, and not every time we want to do any load process. It's also WAY faster to read from memory than from disk, the difference becomes greater on spin-drives and network locations. Having all the information in one file also means that we only have to load from one source, and not two (which we'd then have to merge) which is far faster and leaves only one point of potential failure instead of two. There is also messiness regarding game entries in your library NOT in our database, or entries that started off NOT in it and then since were added and connected, the fact that metadata came from other sources than just this file (like MAME ini files, storefront APIs, etc), and more...

There are definitely things we've identified as things we'd do differently in the future if we had the opportunity, as well as some nice have things that would make certain tasks easier for both user and dev. As time allows we push the app towards those goals, but some of those legacy decisions are rooted DEEP in this code base and so migrating to our newer desires can often time being a MUCH larger and complicated task than you may assume and so get pushed off for other area's of the app we can improve in a more timely manner (that also typically have far larger impact on the overall user experience).

Quite a large number of the performance bottlenecks have been identified and I've offered several solutions to combat, but as the only developer actively working on the PC version of LB there is only so much time in the day and so we have to really triage where and what I do in the codebase for maximum impact to everyone.

[Rinusch]

Good morning!

Thanks for this insight. I now better understand the ideas behind the decisions.
I still wonder how many users actually (implicitly) expect common metadata fields once scraped for their games never to get updated automatically anymore afterwards. For instance, the number of players for a game. I have sometimes corrected entries in the online DB and indeed noticed my game's local metadata didn't get updated to reflect those changes, while the online DB did get updated. To me, this is unexpected and this is what I mean by 'how many users implicitly expect..'.
Custom, user-defined fields on the other hand shouldn't get overwritten. And I can also imagine particular users wanting full control over their local metadata which they may initially import and then maintain themselves. Maybe they could then explicitly indicate that in their settings, or even 'lock' individual fields.
But I also understand the technical point of having to get your data from multiple sources, and I don't know what the potential gains in terms of memory usage, start-up time and performance would be if say 95% of the users would want their metadata to be maintained automatically. If this would be a serious potential gain, it could be a consideration for the future.
But I appreciate you're the only developer actively working on the pc version (also taking your time to carefully respond on this forum which is great!) and understand the struggles..

Still, I have 2 more questions which I hope you can briefly respond to:

1) I see no option to only update metadata for all- or selected games (not only star ratings) and not touch my current media files. Am I correct? I've tried the 'download and replace all existing metadata' option, having unchecked all media types, but it won't update my metadata from de LB DB in that case.

2) I have manually added a large number of media files for games which don't have those files either in your media database or Emumovies' DB (Amiga media is seriously 'underrepresented' imho). I have for a large part just moved those files into the corresponding media folder in de LB folder structure. Although the file names of those media files often don't match the game names very well, they got recognized and were linked to the right games automagically. Now, from the filenames I'm unable to match a media file to a particular game, but how does Launchbox manage to do so? There's no xml file I can find which holds links from games to their media files. So it has to be done by certain logic applied at runtime? If so, what logic is this?
Can I force Launchbox to bulk-rename my media files to correspond with the game names? When I manually rename a game (when editing meta data), the corresponding media files get renamed. Is there a trick to bulk-rename my files?

Thanks and regards!

[C-Beats]

2 hours ago, Rinusch said:

1) I see no option to only update metadata for all- or selected games (not only star ratings) and not touch my current media files. Am I correct? I've tried the 'download and replace all existing metadata' option, having unchecked all media types, but it won't update my metadata from de LB DB in that case.

As far as I'm aware this should update the metadata. If you're not seeing it do so please file a bug report

As far as your second question you can try the bulk image import in the Tools Menu. It has similar logic to the app regarding ability to tie an image to a game and I believe it does rename the file more appropriately as part of that import process.

Regarding the logic used it's FAIRLY simple. We match images based on following criteria:
The image filename is an exact match to the ROM filename (minus the extension)
The image name is the same as the game title (invalid filename characters are turned into underscores for this logic)
The image name is the same as the game title with a numeral suffix (ie Mario Bros-01.png)
The image name is the same as the game title with a ID and numeral suffix (the ID is the game's ID and is used when the same platform has multiple games by the same name)

[Rinusch]

Thanks! I'm really sorry, but I can't find anyting like 'bulk image import' in the Tools menu or anywhere else. Am I missing something?
And by the way, will this possible solution also go for video and audio media files or only image media as the name of this option suggests?

Regards!

Edited June 11 by Rinusch

[C-Beats]

19 minutes ago, Rinusch said:

Thanks! I'm really sorry, but I can't find anyting like 'bulk image import' in the Tools menu or anywhere else. Am I missing something?

Tools > Image Packs > Import

[jophran]

17 hours ago, C-Beats said:

As far as I'm aware this should update the metadata. If you're not seeing it do so please file a bug report

As far as your second question you can try the bulk image import in the Tools Menu. It has similar logic to the app regarding ability to tie an image to a game and I believe it does rename the file more appropriately as part of that import process.

Regarding the logic used it's FAIRLY simple. We match images based on following criteria:
The image filename is an exact match to the ROM filename (minus the extension)
The image name is the same as the game title (invalid filename characters are turned into underscores for this logic)
The image name is the same as the game title with a numeral suffix (ie Mario Bros-01.png)
The image name is the same as the game title with a ID and numeral suffix (the ID is the game's ID and is used when the same platform has multiple games by the same name)

Correct me if i'm wrong, but I think that text between parenthesis get ignored when matching, recently I've been strugling with adding manuals to games that have the same name but are indeed different under the same platform, for example Airwolf and Fantasy Zone in NES

[C-Beats]

5 hours ago, jophran said:

Correct me if i'm wrong, but I think that text between parenthesis get ignored when matching, recently I've been strugling with adding manuals to games that have the same name but are indeed different under the same platform, for example Airwolf and Fantasy Zone in NES

While that's been true in the past it's not 100% accurate currently. If there is a value in a parens we see if that value in parens is also present in the title of the DB game and if so use that as a first round match, if it can't be found THEN it's discarded. It helps us identify which game the ROM is a match to for platforms that had multiple titles for the same game.