Jump to content
LaunchBox Community Forums
Z3R0B4NG

No-Intro / REDUMP .dat compatibility with LaunchBox Database

Recommended Posts

The Problem:  
The Launchbox GamesDB Naming scheme doesn't always match what No-Intro or REDUMP use for filenames. Some rules in the naming convention are drastically different,
some files may have minor spelling errors or include or not include some words (if somebody uses the old "GOOD" sets its much worse with spelling errors)... Leading numbers in filenames are a huge pain to deal with.  
LaunchBox doesn't detect the correct DB entry for games when names don't match, even EmuMovies doesn't always have identical naming so some games that are covered over there are still not scraped while importing here.  
 

How i deal with it for now:  
I always add the filenames of not detected roms to the LBDB as alternate names as i stumble upon them, if i or someone else imports the same rom again they will hopefully be detected properly next time and i hope that i'm not the only one doing this.  
It would be slightly more helpful if there were options like No-Intro or Redump in the drop down menu for the language selection. Or maybe just some universal "ROM DETECTION" option.  
Just so the next guy working on that same entry doesn't delete it again thinking it doesn't match our naming scheme or is a spelling error, but understands that it has a reason for being there.  


Feature Request:  
BEST would be of course if the Launchbox Devs could find a way to utilize the popular .dat files and somehow merge them with the LBDB.  

Result 1: better match accuracy, as most users will be using No-Intro and Redump sets anyway (GOOD? TOSEC?) and use the .dat files to build their collections.  

Result 2: as the hash values are part of the .dat files, it should be possible to not only match by filename but by hash value as well, which will be 100% accurate instead of the guesswork that sometimes goes into matching by name.  
(as i understand it, LaunchBox would not even need to run a hash check on every file, which could take a lot of time, AFAIK the crc values are included in zip and rar files which makes the scanning process a lot faster).  


additional thoughts:  
The .dat files would occasionally require to be updated as No-Intro and Redump are of course ongoing projects that get new dump data on a daily basis (maybe the dev's can figure out a way to automate this, maybe some cooperation with the various projects is possible or maybe allow users to upload .dat files unless that could break stuff to be as hands off as possible).  

There would be no need to clean up old data or bad dumps or even hacked dumps (fan translations etc.) as people might still be using outdated sets (GOOD) and it is not LaunchBoxes job to keep the Users collection clean or up-to-date but to find matching metadata and pretty pictures, so only the new data would need adding.  
The initial matching between .dat and LBDB could be done by the existing algorithm that matches by name, what isn't matched would need to be manually linked, this is something the users could do if a web interface is added in form of a new Tab in the LaunchBox Database moderation interface.  

I think it should be pretty important for the LaunchBox Dev's to find a way to do this somehow as it would effectively increase the match rate to as close as 100% as possible (i mean not every game was dumped in a .dat file yet, not every dumped game or media has a matching LBDB entry etc.).  

...i can only imagine it should be easier to implement this (maybe more tedious) than to deal with the mess that importing MAME was, i have no clue how anyone can even make sense of that stuff. 

This stuff just keeps spinning in my head, i would love to hear what you guys think. Would this be doable? Are there some big roadblocks i'm not aware of?

  • Like 1

Share this post


Link to post
Share on other sites

The way the scraper works currently is that it ignores anything in brackets, so for example "Wave Race 64 (USA) (Rev B)" the (USA) and the (Rev B) are ignored and just the game name "Wave Race 64" is used by the scraper. So as long as your roms arent stupidly named then they will match and import just fine, i could probably count on one hand the games that failed to match to the database when using No Intro sets. The No Intro naming scheme is what is used as a base for the scraper already so that set should have no issue importing. Hacks and prototypes are a different issue as obviously they are not official games so they are always going to be somewhat difficult to match with media. To be honest the way it is currently done is probably the best and most user friendly way to handle it, we dont want to have users have to mess with .dats at all really, a user should just be able to add a folder/files and let Launchbox sort it all with out the need for 3rd party solutions. 

Share this post


Link to post
Share on other sites

I do very much appreciate the effort of adding no intro alternate names, as that will ultimately improve the search for others in the future. That said, it seems to be very rare as of current for a game not to be matched properly to the DB, so I don't consider it to be a priority for the moment, at least. For example, my N64 platform has 100% matched games, while only 5 of my 617 GameCube games weren't matched on initial import. I recognize that the accuracy does go down for hacks/prototypes/etc., but that's usually mostly because of them being missing from the games database.

Eventually I can look to add the No-Intro names to the games database metadata (that is actually something that I had not previously considered doing), but it will likely be a while as I have a lot of other things that are higher up on the priority list. Still, thanks for the proposal; that is certainly a good idea. :)

Share this post


Link to post
Share on other sites

Not quite correct @neil9000  the No-Intro naming scheme for example has "Legend of Zelda, The" while LBDB has it as "The Legend of Zelda", small stuff like this can already throw off the matching process.  
(this stuff i usually end up adding as alternate title and is easy enough to figure out quickly on your own, just tedious to go through if there are many like that in a system)  

Redump often includes Disney or Disney-Pixar as the first bit of a games name which then often won't match LBDB. (just working through PS1 games has a ton of none matches just because of this alone).  

What i didn't consider and just realized is that many of my mismatches are because i also import Euro and maybe worse German versions, those are less likely to be matched because the naming can differ wildly and you probably have way less German users that moderate the DB than US based users.  

V-Rally 2 on the PS1 for example was "Need for Speed: V-Rally 2" in the US because it was published by EA.... (just for lols: the Dreamcast version of the game was published under the Test Drive Series, again by a different Publisher, i just find that funny as those Series have usually been seen as rivals.).  That is the kind of stuff that is a bit harder to find matching entries for and even requires checking Wikipedia pages sometimes (in fact there were 2 entries for the same game and i needed to merge those and delete the obsolete Euro one after that, that one will be matched properly next time, you're welcome 😜 ).  

Maybe i just expected if the full No-Intro / Redump dats were worked with instead that all region variants would automagically be covered by that approach as well.  
But yeah, somebody would probably still need to manually match the odd stuff (but at least more eyeballs would be on it and we would have a list of stuff that is missing to work through).  

  • Like 1

Share this post


Link to post
Share on other sites

@Z3R0B4NG I'm 100% agree with you. I lost many hours to change manually rom names in my Launchbox to be able to get the missing artworks or movies from Emumovies. So, my main problem is not so much with the LB database but with how the scrap is done with emumovies. It seems that alternate names are no use for that, LB use only the main name against Emumovies.  Maybe if you have only US ROMs your experience will be completely different. I will use european version as much as possible because I want french language. I post several post about that last week without any answer and I'm pleased to see I'm not alone and not completely dumb! Finally, update the LB database with No-intro and Redump dats would be a clever move because many collectors use them. Why lost so many time to add alternate names, one by one, in the database when it could be so easy to do that in bulk? Anyways, great work guys, I love to be part of this community!

Edited by Fablog

Share this post


Link to post
Share on other sites

I would like to add I also have random scrapes fail because of an ending like ", The" at the end of rom names, or anything that ends with a comma, then another word or two. This type of rom name has always been problematic with matching the LB DB and/or Emumovies. I even wrote a file renamer script along with my own audit program that shows me very granular results as to missing artwork, manuals, music and/or videos. Anything missing is almost always the difference in a rom beginning, or ending in "The". There are other random missing scrapes, but the one outlines here seems to be the most prevalent with my files.

Finally, I also use No-intro files when possible, so loading the LB DB with those names would make me a very happy man. :)

Share this post


Link to post
Share on other sites

For Videos you have an option "Tools > Audit [System Name]"  

that will spit out a spreadsheet, you scroll a bit to the right and click once on the column "video path" and it sorts all the empty entries to the top, now you got all the games with missing videos neatly sorted to the top.  

I then use the EmuMovies FTP Server and look for the missing videos by hand (you need to have donated to them to get access to their Server). ...well i did this ONCE and it worked well enough.  
You can do the same with music files i guess. But i don't see manuals in that audit list.  

LaunchBox is awesome, it finds like 90 - 99% of the good stuff, it just gets tedious for the last few % where it doesn't, exponentially so with the growing size of the collection.

Share this post


Link to post
Share on other sites

It's exactly what I do but it's tedious and you have to change the ROM title in your LB if you want to import the missing videos after that. Like you said, the last few % take me too much time if I want a collection well organized and presented.

Edited by Fablog

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×