majormajor0

I chose SQLite for portability and deployability. I could never figure out how to deploy SQL without an install. But also, I've long thought (as I mentioned) that the real value in the project is in the DB. And having it in SQLite makes it easier, I think, for people to get at it. I've been meaning to add Moby Games and GameFAQs. I believe both of them have good region data, which is a weak spot in most DBs. But unless I get some outside interest in this, I'm not sure big further effort is justified. I also thought that the cross-DB maps might be useful here to punch up the LaunchBox GamesDB. With the tools I've written, one could, in a matter of hours, basically absorb 3 other DBs into Launchbox. By the way, the front end contains tools to rapidly compare databases and find matches. It standardizes the titles by tossing meaningless characters and spaces, and replacing probable inconsistencies (e.g., IV = 4). Then for unmatched entries, it takes the Levenstein distance and sorts out the nearest possible matches so you can go in and look for manual matches. I think this stuff is highly useful, but, again, more so for the DB than for a front end user.

Well, after much twiddling, this is kind of done to a beta level. The front end is what it is--it is no launchbox, to put it mildly. But I believe the database could have value for other developers (see below). After a couple of false starts, I scrapped the website, but the GitHub is here (https://github.com/MajorMajor0/Robin). It includes the database as a SQLite file. There is no install. The driving purpose of this project is to sort and eliminate the mountains of junk in the ROM world and boil it down to unique, playable, non-junk games. Database features: - 25,960 seperate releases sorted into 12,100 unique, playable, non-junk games - All platform data up to 4th gen - Essentially limited to sprite-based consoles and handhelds - This is because later gen games are impractically large - I have tools to pretty rapidly add any requested platform, especially a platform in Datomatic. - MAME data for .195. - Note that this is limited to playable games and their parents, in keeping with the purpose of eliminating junk--i.e., games not in the DB should be assumed junk. - Note also that the MAME version can be easily changed--up or down given access to a MAME exe file. - Based on Datomatic (no-intro) for release, region and title data--this is after a lot of painful research to choose the cleanest data - Metadata pulled from GamesDB, GiantBomb, LaunchBox, and OpenVGDB, cleaned and merged - The DB as a by-product contains matches between all of these databases, as well as Datamatic - A major feature that would be hard to find anywhere else is the cross-reference here between ROM checksums and games in these major databases - All standard metadata found in any of these DBs - Custom metadata that took a lot of effort to putting together, but is part of the junk-filtering process. - Junk: 509 games - Adult: 230 games - Not game: 710 games (for instance, TI-89) - Mess machine: 703 games (not a standard video game, for instance LED hand held or arm-wresting machine) - Multiple releases (clones) are gathered into games in order to bundle and hide duplicates Front-end features - Goes through your mountains of ROMS, tosses garbage and duplicates, and sorts and identifies the good stuff (nothing is deleted) - Shows what you have and don't have - Filtering in milliseconds on 13 metadata properties - Autofiltering shows only valid remaining filter choices - Real-time, very fast text filtering - Display by individual release, or grouped into unique games - Display platforms and emulators - Update database with latest and greatest from online DBs, search for new matches - Rudimentary CLRMAMEPro style functionality. - Orders of magnitude faster than CLRMAMEPro or ROMCenter - Not as good as either of these excellent programs, yet, since it is just started. Still, faster. So, again, the front-end is what it is, but I think the database could be a resource for developers.

Jason, This metadata file you provide is fantastic--it fits right in with what I'm doing with the other DBs, which is to periodically cache the relevant parts in a local SQLite--much appreciated. I don't see how to get an image URL from it, though. The DB I am working on is my own little hobby project. I don't have it in front of me, but it looks superficially like your much more professional effort, and in fact, had I seen yours before I started I would probably have never started. The philosophy is to sort through all of mountains of available data and pare it down to official releases, group them by games, then filter by metadata. I am starting from Datomatic and the MAME DB to discard hacks and unplayable games. So while the front end will acknowledge and catalog all the releases of Excitebike, for instance, it indents them together under one game and flags one of the releases as preferred for launching. I'm also cutting it off at 4th generation consoles in order to bound the project. The other component to it is tools to map the games to online DBs to gather as much available metadata as possible. Part of that is making it so if you are playing a European release, you are looking at European boxart. Right now I have online DB matches for about 90% of non-MAME (Datomatic) Games/Releases, but only about 30% of MAME machines. I was very encouraged, recently, that when I refreshed all of the cache after a few months, there were hundreds of new hits. Right now I have been tearing my DB down and rebuilding it with a mildly different structure. When it's back together, I'll post it for public consumption in case it's of any use. It may well be useful, once I add a column for Launchbox, because it will be mapping your DB to 3 other DBs from then on.

Oh, well, Mr. Carr got in there while I was typing.

I interpreted silence as consent. It's not my highest priority right now, since I'm already scraping 3 databases, but when I get to it, I'll add launchbox. My plan is to add a second or two delay to protect the site from my app turning into a DDoS attack. I very much suspect that most of the problems with GamesDB stem from them not enforcing limits over there.

Am putting the finishing touches on (yet another) front-end right now. So many online DBs with so many problems... I'd like to add this site as a scraping source. Just poking through in a browser, I think that would be straightforward, even without an API. Jason, it sounds from your post in April like you are open to being scraped...I wanted to hear what limits you'd put on that, and what you mean by getting a solution going.