No-Intro / REDUMP .dat compatibility with LaunchBox Database

[Z3R0B4NG]

The Problem:  
The Launchbox GamesDB Naming scheme doesn't always match what No-Intro or REDUMP use for filenames. Some rules in the naming convention are drastically different,
some files may have minor spelling errors or include or not include some words (if somebody uses the old "GOOD" sets its much worse with spelling errors)... Leading numbers in filenames are a huge pain to deal with.  
LaunchBox doesn't detect the correct DB entry for games when names don't match, even EmuMovies doesn't always have identical naming so some games that are covered over there are still not scraped while importing here.  
 

How i deal with it for now:  
I always add the filenames of not detected roms to the LBDB as alternate names as i stumble upon them, if i or someone else imports the same rom again they will hopefully be detected properly next time and i hope that i'm not the only one doing this.  
It would be slightly more helpful if there were options like No-Intro or Redump in the drop down menu for the language selection. Or maybe just some universal "ROM DETECTION" option.  
Just so the next guy working on that same entry doesn't delete it again thinking it doesn't match our naming scheme or is a spelling error, but understands that it has a reason for being there.  

Feature Request:  
BEST would be of course if the Launchbox Devs could find a way to utilize the popular .dat files and somehow merge them with the LBDB.  

Result 1: better match accuracy, as most users will be using No-Intro and Redump sets anyway (GOOD? TOSEC?) and use the .dat files to build their collections.  

Result 2: as the hash values are part of the .dat files, it should be possible to not only match by filename but by hash value as well, which will be 100% accurate instead of the guesswork that sometimes goes into matching by name.  
(as i understand it, LaunchBox would not even need to run a hash check on every file, which could take a lot of time, AFAIK the crc values are included in zip and rar files which makes the scanning process a lot faster).  

additional thoughts:  
The .dat files would occasionally require to be updated as No-Intro and Redump are of course ongoing projects that get new dump data on a daily basis (maybe the dev's can figure out a way to automate this, maybe some cooperation with the various projects is possible or maybe allow users to upload .dat files unless that could break stuff to be as hands off as possible).  

There would be no need to clean up old data or bad dumps or even hacked dumps (fan translations etc.) as people might still be using outdated sets (GOOD) and it is not LaunchBoxes job to keep the Users collection clean or up-to-date but to find matching metadata and pretty pictures, so only the new data would need adding.  
The initial matching between .dat and LBDB could be done by the existing algorithm that matches by name, what isn't matched would need to be manually linked, this is something the users could do if a web interface is added in form of a new Tab in the LaunchBox Database moderation interface.  

I think it should be pretty important for the LaunchBox Dev's to find a way to do this somehow as it would effectively increase the match rate to as close as 100% as possible (i mean not every game was dumped in a .dat file yet, not every dumped game or media has a matching LBDB entry etc.).  

...i can only imagine it should be easier to implement this (maybe more tedious) than to deal with the mess that importing MAME was, i have no clue how anyone can even make sense of that stuff. 

This stuff just keeps spinning in my head, i would love to hear what you guys think. Would this be doable? Are there some big roadblocks i'm not aware of?

[neil9000]

The way the scraper works currently is that it ignores anything in brackets, so for example "Wave Race 64 (USA) (Rev B)" the (USA) and the (Rev B) are ignored and just the game name "Wave Race 64" is used by the scraper. So as long as your roms arent stupidly named then they will match and import just fine, i could probably count on one hand the games that failed to match to the database when using No Intro sets. The No Intro naming scheme is what is used as a base for the scraper already so that set should have no issue importing. Hacks and prototypes are a different issue as obviously they are not official games so they are always going to be somewhat difficult to match with media. To be honest the way it is currently done is probably the best and most user friendly way to handle it, we dont want to have users have to mess with .dats at all really, a user should just be able to add a folder/files and let Launchbox sort it all with out the need for 3rd party solutions. 

[Jason Carr]

I do very much appreciate the effort of adding no intro alternate names, as that will ultimately improve the search for others in the future. That said, it seems to be very rare as of current for a game not to be matched properly to the DB, so I don't consider it to be a priority for the moment, at least. For example, my N64 platform has 100% matched games, while only 5 of my 617 GameCube games weren't matched on initial import. I recognize that the accuracy does go down for hacks/prototypes/etc., but that's usually mostly because of them being missing from the games database.

Eventually I can look to add the No-Intro names to the games database metadata (that is actually something that I had not previously considered doing), but it will likely be a while as I have a lot of other things that are higher up on the priority list. Still, thanks for the proposal; that is certainly a good idea.

[Z3R0B4NG]

Not quite correct @neil9000  the No-Intro naming scheme for example has "Legend of Zelda, The" while LBDB has it as "The Legend of Zelda", small stuff like this can already throw off the matching process.  
(this stuff i usually end up adding as alternate title and is easy enough to figure out quickly on your own, just tedious to go through if there are many like that in a system)  

Redump often includes Disney or Disney-Pixar as the first bit of a games name which then often won't match LBDB. (just working through PS1 games has a ton of none matches just because of this alone).  

What i didn't consider and just realized is that many of my mismatches are because i also import Euro and maybe worse German versions, those are less likely to be matched because the naming can differ wildly and you probably have way less German users that moderate the DB than US based users.  

V-Rally 2 on the PS1 for example was "Need for Speed: V-Rally 2" in the US because it was published by EA.... (just for lols: the Dreamcast version of the game was published under the Test Drive Series, again by a different Publisher, i just find that funny as those Series have usually been seen as rivals.).  That is the kind of stuff that is a bit harder to find matching entries for and even requires checking Wikipedia pages sometimes (in fact there were 2 entries for the same game and i needed to merge those and delete the obsolete Euro one after that, that one will be matched properly next time, you're welcome ? ).  

Maybe i just expected if the full No-Intro / Redump dats were worked with instead that all region variants would automagically be covered by that approach as well.  
But yeah, somebody would probably still need to manually match the odd stuff (but at least more eyeballs would be on it and we would have a list of stuff that is missing to work through).  

[Fablog]

@Z3R0B4NG I'm 100% agree with you. I lost many hours to change manually rom names in my Launchbox to be able to get the missing artworks or movies from Emumovies. So, my main problem is not so much with the LB database but with how the scrap is done with emumovies. It seems that alternate names are no use for that, LB use only the main name against Emumovies.  Maybe if you have only US ROMs your experience will be completely different. I will use european version as much as possible because I want french language. I post several post about that last week without any answer and I'm pleased to see I'm not alone and not completely dumb! Finally, update the LB database with No-intro and Redump dats would be a clever move because many collectors use them. Why lost so many time to add alternate names, one by one, in the database when it could be so easy to do that in bulk? Anyways, great work guys, I love to be part of this community!

Edited September 25, 2018 by Fablog

[dragon57]

I would like to add I also have random scrapes fail because of an ending like ", The" at the end of rom names, or anything that ends with a comma, then another word or two. This type of rom name has always been problematic with matching the LB DB and/or Emumovies. I even wrote a file renamer script along with my own audit program that shows me very granular results as to missing artwork, manuals, music and/or videos. Anything missing is almost always the difference in a rom beginning, or ending in "The". There are other random missing scrapes, but the one outlines here seems to be the most prevalent with my files.

Finally, I also use No-intro files when possible, so loading the LB DB with those names would make me a very happy man. :)

[Z3R0B4NG]

For Videos you have an option "Tools > Audit [System Name]"  

that will spit out a spreadsheet, you scroll a bit to the right and click once on the column "video path" and it sorts all the empty entries to the top, now you got all the games with missing videos neatly sorted to the top.  

I then use the EmuMovies FTP Server and look for the missing videos by hand (you need to have donated to them to get access to their Server). ...well i did this ONCE and it worked well enough.  
You can do the same with music files i guess. But i don't see manuals in that audit list.  

LaunchBox is awesome, it finds like 90 - 99% of the good stuff, it just gets tedious for the last few % where it doesn't, exponentially so with the growing size of the collection.

[Fablog]

It's exactly what I do but it's tedious and you have to change the ROM title in your LB if you want to import the missing videos after that. Like you said, the last few % take me too much time if I want a collection well organized and presented.

Edited September 26, 2018 by Fablog

[NahuelDS]

I'm still waiting for this feature to properly use LaunchBox.
Don't get me wrong, LaunchBox is really nice... but without DAT implementation, it's impossible for me to automate the tedious task of managing my collection.

Personally, my main issue with LaunchbBox is with the way it imports roms... it still doesn't merge multiple region roms (e.g. "Pocket Monsters - Aka (Japan) (SGB Enhanced)" doesn't combine with "Pokemon - Red Version (USA, Europe) (SGB Enhanced)"). Judging by the feature polls, both the community and the developers prioritize aesthetics over this type of advanced usability.... so I can assume a feature like this won't be implemented. ?

A couple of years ago, I proposed a way to read the contents of a zip/7z file, and choose which rom you wanted to launch from withing the zip file. That way, you only have to import your merged set, which consists on only 1 zip file per game, with all the variants of the same game inside (regions, revisions, beta, etc). But it seems nobody cares about that ?... I always find the same answer "just use 1 rom per game" ?‍♂️

It's been 3 years since I discovered LaunchBox. At first I was really excited... but the fact that I STILL cant manage my collection the way I want, it made stop using this frontend.

[Lordmonkus]

18 minutes ago, NahuelDS said:

I STILL cant manage my collection the way I want

Make your own frontend then.

[Lordmonkus]

28 minutes ago, NahuelDS said:

Judging by the feature polls, both the community and the developers prioritize aesthetics over this type of advanced usability

Here is the list of poll items that received enough votes to get prioritized for development until the next poll. I have bolded the ones that are function over aesthetic. As you can see it is over half. This does not include things that get added in along the way.

BigBox integrated theme manager and downloader with update notifications.
Mame overhaul (updated metadata, improved filtering, importing clones into additional apps).
Pause screen with manuals support.
Game startup launch screen with progress bar and improved multiple instance startup prevention. (I bolded half because half of this function and half is aesthetic)
Add Retroarch core to the Launch With menu (nested under Retroarch).
Retro Achievements further integration (Gamer score, leaderboards)
Options to filter out games that are missing images/media.
Easier Retroarch command line integration drop down with core detection.
Control definitions with overlays / icons (Display the controls for a game / platform).
Proper support for Pinball (Media and emulators).
Built in hide mouse cursor features.
BigBox per platform background music and sounds.
Custom default game images for each platform.
Custom nested folders / nested filters (support for multiple levels).

[Z3R0B4NG]

8 hours ago, NahuelDS said:

my main issue with LaunchbBox is with the way it imports roms... it still doesn't merge multiple region roms (e.g. "Pocket Monsters - Aka (Japan) (SGB Enhanced)" doesn't combine with "Pokemon - Red Version (USA, Europe) (SGB Enhanced)").

A couple of years ago, I proposed a way to read the contents of a zip/7z file, and choose which rom you wanted to launch from withing the zip file. That way, you only have to import your merged set, which consists on only 1 zip file per game, with all the variants of the same game inside (regions, revisions, beta, etc). But it seems nobody cares about that ?... I always find the same answer "just use 1 rom per game" ?‍♂️

Well, it is supposed to merge different region roms, that is literally one of its functionalities, it just relies on the LaunchBox Database and especially when it comes to JAP region names it is very incomplete ...the reason for that is simply that most people using the App are US or EU based, or so i would assume, most of us probably just don't use JAP roms to begin with because most don't speak the language, resulting in comparatively poor support for JAP region titles in the database.  
With EU roms it is not that bad but there is the occasional odd one out, France and German specific versions are again a bit worse.  

.dat support would improve all that, but there would still need to be some hands on work with games that have wildly different titles, software doesn't automagically know that "Pocket Monsters" is the same as "Pokémon" (and yes the apostrophe above the é is also something that WILL throw off detection, i went through a lot of pokemon titles and added names for that reason, i'm sure i didn't get all but it should be a bit better as of 3+ months ago ...and i don't even play that stuff!).  

merged sets...?  
how would you end up with merged sets anyway?  
are you using the "GOOD" set? don't do that, use No-Intro.  
With No-Intro you will not end up with merged sets.  
This is probably not helpful, but there are Emulators who work with merged sets (bSNES i think gave me a selection screen once?), as long as you use one of those you can select the specific rom in the emulator and don't need LaunchBox to do that for you.  

Either way, i can imagine some edge cases where such a functionality could be useful.

Edited January 4, 2019 by Z3R0B4NG

[self_slaughter]

First post, so Hi and thanks for the great launcher!

Sorry if this is considered a necro, but google led me here when trying to see if I could make use of dats to ease importing/updating my sets in launchbox.

No-intro does actually supply (constantly updated) dats with the list of parent/clone games and their regions to make said "merged sets". Very useful information that would take a lot of the guess work out of various language barriers etc and allow launchbox to do a far nicer job of merging some games.

The amount of work they put into that kind of stuff is insane. Much more to them than just filenames and hashes. Seems crazy to not make use of that information somehow. Either feeding the dats into launchbox directly, or just to run the names against the database occasionally. 

If you could take it a step further and actually use launchbox to update your sets and fix any changed entries in the launcher at the same time, that would be some next level stuff. (But obviously a massive job that won't happen). Still, nice to dream. hahaha
 

Example (Text Version):

6087 - Learn with Pokemon - Typing Adventure (Europe)
	5669 - Battle & Get! - Pokemon Typing DS (Japan)
	6274 - Apprends avec Pokemon A la Conquete du Clavier (France)
	6380 - Impara con Pokemon - Avventura Tra i Tasti (Italy)
	6401 - Lernen mit Pokemon - Tasten-Abenteuer (Germany)
	6533 - Aprende Con Pokemon - Aventura Entre Las Teclas (Spain)

Example (Straight out of the Parent/Clone DAT):

	<game name="Learn with Pokemon - Typing Adventure (Europe)">
		<description>Learn with Pokemon - Typing Adventure (Europe)</description>
		<release name="Learn with Pokemon - Typing Adventure (Europe)" region="EUR"/>
		<rom name="Learn with Pokemon - Typing Adventure (Europe).nds" size="134217728" crc="73d4ad7a" md5="99655f755f06c3ca400bb9e080d25231" sha1="45184ec8998ca6fe4641df154820ad6923b7db1f" status="verified"/>
	</game>
	<game name="Apprends avec Pokemon A la Conquete du Clavier (France)" cloneof="Learn with Pokemon - Typing Adventure (Europe)">
		<description>Apprends avec Pokemon A la Conquete du Clavier (France)</description>
		<release name="Apprends avec Pokemon A la Conquete du Clavier (France)" region="FRA"/>
		<rom name="Apprends avec Pokemon A la Conquete du Clavier (France).nds" size="134217728" crc="bd728697" md5="ee099bd969beccb7f61a91767abf221f" sha1="6899d6265b47230d20c608bed105ea0fa92b4a08"/>
	</game>
	<game name="Aprende Con Pokemon - Aventura Entre Las Teclas (Spain)" cloneof="Learn with Pokemon - Typing Adventure (Europe)">
		<description>Aprende Con Pokemon - Aventura Entre Las Teclas (Spain)</description>
		<release name="Aprende Con Pokemon - Aventura Entre Las Teclas (Spain)" region="SPA"/>
		<rom name="Aprende Con Pokemon - Aventura Entre Las Teclas (Spain).nds" size="134217728" crc="c91de98e" md5="f2e338517a0731f915fd0d8163c1cd46" sha1="0b6411b9dbf64b8724e663e85042a5bc3b16a9e3"/>
	</game>
	<game name="Battle &amp; Get! - Pokemon Typing DS (Japan)" cloneof="Learn with Pokemon - Typing Adventure (Europe)">
		<description>Battle &amp; Get! - Pokemon Typing DS (Japan)</description>
		<release name="Battle &amp; Get! - Pokemon Typing DS (Japan)" region="JPN"/>
		<rom name="Battle &amp; Get! - Pokemon Typing DS (Japan).nds" size="134217728" crc="b5d625d9" md5="51cb587281bf9f06ee56823c1143dc5d" sha1="ad5a5347028fc40c9384e7f32039e09d8697875c" status="verified"/>
	</game>
	<game name="Impara con Pokemon - Avventura Tra i Tasti (Italy)" cloneof="Learn with Pokemon - Typing Adventure (Europe)">
		<description>Impara con Pokemon - Avventura Tra i Tasti (Italy)</description>
		<release name="Impara con Pokemon - Avventura Tra i Tasti (Italy)" region="ITA"/>
		<rom name="Impara con Pokemon - Avventura Tra i Tasti (Italy).nds" size="134217728" crc="8e0044a4" md5="e6712f080b391012a49517d1290c81e4" sha1="1c104f907950454e815b460345c4b80fd68f1ee2"/>
	</game>
	<game name="Lernen mit Pokemon - Tasten-Abenteuer (Germany)" cloneof="Learn with Pokemon - Typing Adventure (Europe)">
		<description>Lernen mit Pokemon - Tasten-Abenteuer (Germany)</description>
		<release name="Lernen mit Pokemon - Tasten-Abenteuer (Germany)" region="GER"/>
		<rom name="Lernen mit Pokemon - Tasten-Abenteuer (Germany).nds" size="134217728" crc="7641bf1b" md5="595080adf211f6054a83d363243d326d" sha1="d26690a6f9fe81afc81008818b0e50f0ed2d6c35" status="verified"/>
	</game>