Changes dispute process and a few concerns

[Captain Star Paw]

I'm relatively new here, but I am very active in contributing alternate ROM set titles, especially with respect to the No-Intro database. I have a few questions. Also, please excuse me if I am being profoundly dumb, or if I have missed the obvious.

First question. Is there a way to see who/why someone rejected a particular contribution? Or is the case where-by if no reason is given, the rejection is recorded without the possibility of dispute?

I have been slightly frustrated by people not properly cross-examining my name contributions, because for example, one or two people seem to be rejecting names and titles with commas in them such as "Siedler, Die". They are also not providing a reason why. If they were to take a second to check https://datomatic.no-intro.org/?page=show_record&s=40&n=0401 or searching at the SPS http://www.softpres.org/games for example, they would find out that my name submission is correct.

This is a naming standard convention that also applies to TOSEC https://www.tosecdev.org/tosec-naming-convention

Second and third question, which is also a suggestion.

Does Launchbox use a hash table?

Are there any plans to collate all the ROM DAT names from popular DAT collections so we can link names to the Launchbox database?

I feel that the process I have been following thus far is less about submitting a name, but rather, saying that "this" ROM is from "this" DAT collection, where the name information is already pre-set. It would be great to see the Launchbox database implement a more robust way of recognising ROMs.

Also, I feel that this ability should be tied directly into the front-end application interface, perhaps by offering a wizard that asks the user 1) Are you a contributor? or 2) Are you just here to game? Optimising the user experience by tailoring the options available.

Edited March 12, 2020 by Captain Star Paw
Added TOSEC naming convention source

[Thanatos_Prime]

If you go into your changes under My Account, Changes Status on the item that was rejected on the right side there is an i icon that you can click on to find out why it was rejected. In most cases I would suspect that you are using dashes in the game names. Per game submission rules, no dashes as separators should be used in game names, substitute : in all such cases. So instead of "This is the Game - Second Edition" you should be using "This is the Game: Second Edition". Additionally, you should remember that the names currently in the DB are probably there for a reason, so while I support this naming standard with the dash to colon changes above so that it supports our database, you should take the time to make the titles you are overwriting as alternate game names so that people using the ROM name that got it into the DB to begin with still having matching in the future. 

[Z3R0B4NG]

On 3/9/2020 at 4:01 PM, Captain Star Paw said:

First question. Is there a way to see who/why someone rejected a particular contribution? Or is the case where-by if no reason is given, the rejection is recorded without the possibility of dispute?

You can only see in your change status window why somebody rejected a change and only if the submission was actually rejected. If one guy rejects and 3 wink it through, you will not see his reason, which always has me wondering why.

On 3/9/2020 at 4:01 PM, Captain Star Paw said:

 people seem to be rejecting names and titles with commas in them such as "Siedler, Die".

Because this is against the naming rules of the LaunchBox Database.  
We are supposed to use "common sense" for alternate names but some people stick to the rules a bit too much.  

LaunchBox is intelligent enough when importing roms to make "Die Siedler" out of "Siedler, Die" and it searches in the Database for that name, not for the filename.  
There are some exceptions where LaunchBox gets confused ("Sieder, Die - Gold Edition, Die" may as theoretical example be changed into "Die Sidler, Die: Gold Edition" which is of course wrong, but THAT is what you then should add as alternate name so it gets detected), ultimately you should not be working of the filenames but of the names that LaunchBox has imported.  
And generally you should only add a alternate names if LaunchBox failed to find the DB entry, you can "arrange by" in LaunchBox after "LaunchBox Database ID", that way you can work your way through only the stuff that has not been found. (that is how i always do that anyway and i've added a ton of No-Intro names myself already).  

On 3/9/2020 at 4:01 PM, Captain Star Paw said:

Does Launchbox use a hash table?

Sadly no, i have suggested one or two times myself to use the No-Intro / Redump / TOSEC .dat files.  

On 3/9/2020 at 4:01 PM, Captain Star Paw said:

Also, I feel that this ability should be tied directly into the front-end application interface, perhaps by offering a wizard that asks the user 1) Are you a contributor? or 2) Are you just here to game? Optimising the user experience by tailoring the options available.

eeeeh... i'd rather not have the normies that just use the program spam wrong submissions into the DB.
Keep in mind it also scrapes media from EmuMovies and people might not understand the difference if they don't look directly at the LBDB entry.  
We have enough in-fighting in the database as it is, some times i wish we had a few people with Admin rights or super moderators who could lock certain entries.
Like naming for the Pokemon games is a constant and annoying back and forth between "Pokemon" and "Pokémon" and we constantly have to defend those.  

I've recently noticed with C64 games that there are a lot of games (most of it Homebrew) that have identical names, LaunchBox will only see the first game in the database that has an identical name for the same system.  
So i went through a lot of them recently and gave them unique names, like adding the publisher after the name "Aliens (CP Verlag)" and "Aliens" as alternate name, that way all 5 "Aliens" games will show up but it will also show the unique name with enough information so you can decide what game to click on (that is the point where hash values would be super convenient).  
Now of course that is the "common sense" approach and not the "stick 1:1 by the rules"-rule approach, so we constantly have people that try to revert those changes (changes that by the way 3 other moderators have accepted).  

There is no way of telling people about this issue, i can't see who makes the changes so i can't direct message them on the Forum, there is no way for them to see the "Reason" why the change was made so of course they think its wrong.
So we get stuck in this dumb loop of renaming stuff and defending our changes by rejecting changes from people that didn't get the memo... and chances are that the current 3 moderators don't know the reasons either and just wink the changes through.  

So ... eeeeh, i'd rather keep the little barrier up and not have that implemented in the App.  

Edited May 8, 2020 by Z3R0B4NG

[kurzih]

I without any doubt agree with you on the unique naming part and usage of common sense. I also just posted about this issue we were talking about earlier in the troubleshooting:
 

it's a very Déja vu feeling when we've just fixed something a few weeks ago and badaboom, someone tries to unfix them. Of course I don't believe that it's meant to be on purpose to mix things up. Thankfully there are some great experienced database moderators doing a great job every day and motivated new ones as well - and the contribution is stellar, the database is really fantastic. But as you said it only takes 3 new moderators who aren't familiar with all the stepping stones that might turn a few hair grey on the ones who had fixed something before 

[Captain Star Paw]

6 hours ago, Z3R0B4NG said:

LaunchBox is intelligent enough when importing roms to make "Die Siedler" out of "Siedler, Die" and it searches in the Database for that name, not for the filename.  

Not in my experience. Maybe I was using an old version of LaunchBox, but it certainly wasn't intelligent enough to understand the difference; this was my motivation behind submitting the alternate names in the first place.

6 hours ago, Z3R0B4NG said:

Sadly no, i have suggested one or two times myself to use the No-Intro / Redump / TOSEC .dat files.  

Do we have a general consensus with that? If so, do you think it will be a planned feature? Not only would this make it easier to correctly set metadata, but it will encourage people to use properly hashed sets.

What can we do to push for its implementation?

Edited May 9, 2020 by Captain Star Paw

[Thanatos_Prime]

3 hours ago, Captain Star Paw said:

Do we have a general consensus with that? What can we do to push for its implementation?

We really must support this, obviously using alternate names. It makes complete sense. That said, I'm not looking forward to hundreds of extra additions a day for the next month if it can't be handled as MAME was. I vote for No-intro, so I guess that is the next fight.

 

[Jason Carr]

Hey guys, let me clarify a few things. The main game name is important to be named according to the rules of the database. However, half the point of the alternate titles is for flexibility, and they DO NOT need to obey the same rules as the main game title, especially since we're using it to achieve better recognition of ROM files. It doesn't make any sense to reject alternate titles because they don't obey the rules in that case. I appreciate folks like @Captain Star Paw adding them to improve ROM recognition.

Also, I agree that it would be beneficial to integrate the DATs. It has come up before, but it's been a long time. I can see that being a big plus for the database as well. I've got a ton on my plate, but this has been a good reminder; I'll look into the feasibility of all of that here soon. Let me know if you have any suggestions or recommendations in that regard.

[Z3R0B4NG]

8 hours ago, Jason Carr said:

Let me know if you have any suggestions or recommendations in that regard.

I would suggest, just like the alternate names field to let us add hash values to database entries.  
Instead of regions you could have a drop down menu with No-Intro / Redump / TOSEC / GOOD / other.  
There are roms that of course do not come from the common sources or are scene rips (C64 scene for example, the stuff used in C64 Dreams).  
 

Another thing i recently learned is that Dat-o-Matic has Parent/Clone sets for No-Intro, those exist so you can make "1G1R" (1 game 1 release) sets with it, with that data i think it could be possible to make it a bit easier because you should be able to extract every hash for a specific game and group them together right from the .dat file.  

The filenames should match for the largest part with the LBDB already, if you can apply the same algorithms when importing roms into launchbox to add these hash values to DB entries then a good chunk of it should happen automagically, anything that doesn't get matched you could put in a list and post it so we moderators can edit it in.  

That parent/clone .dat isn't a thing for Redump, and i don't know about TOSEC, never used it myself.  

There is an app called "Hash Tab", it adds a Tab in the windows explorer's file properties menu and you can copy/paste and compare CRC / MD5 / SHA-1 hashes easily with it, do recommend (and its free).  

 

The next level crazy idea is, if you already have to deal with .dat files, hash values and check the content of .zip / .rar / .7z files for hash values, you might as well go the extra mile (or extra 10 miles) and add a proper rom manager right into launchbox ... boom who needs CLRMamepro anymore?  
That would be a feature that other front ends don't have at all, not AFAIK. No idea if other users would want that, no idea if that makes the whole idea cooler or just more daunting :D ...good night.

 

//edit:

good morning,  
so i've done some more thinking and a few things would still be problemativ even after perfect hash value recognition is implemented into LaunchBox.  

1) C64 Dreams / eXoDOS  

For starters C64 is a system that is NOT well covered by No-Intro, the three dat files are horribly incomplete and full of games that can't even be emulated because the No-Intro roms include copy protections, so with C64 you want to do the opposite and go with cracked scene released that often do have Intros. Which is what C64 Dreams from Zombweaver does, but even then C64 Dreams does not directly link to the rom files but has come up with some hacky solution with .vbs .bat and .cmd files to launch games with added instructions for retro arch and other stuff (which i don't quite understand).  

Point being you would have to trace the URL to the actual rom file through those vbs and bat files. Adding the hash values of those .bat files directly would be a very messy solution and i don't even know if those files are unique enough to be identified that way (i think a lot of it is actually copy&paste).  

The thing is, C64 dreams has kind of kickstarted this entire discussion because that triggered me to deal with this problem of C64 having one time even up to 8 games with the identical name and LaunchBox only displays the first result it finds. And if that still wouldn't work, even after hash values being a thing... that would annoy the hell out of me.  

eXoDOS has a hacky system of its own that i have not messed with at all.  

Those kind of projects certainly deserve consideration if something can be done to improve rom detection there as well, ...after the .dat files have been dealt with.  

 

2) Nintedo 3DS decrypted roms  

No-Intro has a set for encrypted and decrypted Nintendo 3DS roms, i have to this day not found out how you can decrypt a rom from the encrypted set and get a decrypted rom that matches the No-Intro dat file's hash value.  
All decryption tools that are flying around the internet have different methods of culling trash data from the roms to reduce filesize further, that of course results in practically all roms having different hash values.  
That has been annoying me for a long time now and i got no solution to it.  
Of course the name recognition would still work as fallback here, that is a selfmade problem of the 3DS scene.  

3) Redump multi disc games M3U / CUE / BIN  

You added M3U support recently so you know how that stuff works.  
It isn't ideal for how i do stuff so i still make my own m3u files and import those in LaunchBox.  
So you would again need to follow the path through those text files to the actual rom file through the .m3u and .cue to get to the .bin's hash value .... actually i think the .cue file hash is inside the redump .dat file as well, maybe that would be enough already (and the .cue files are a lot smaller, faster to scan for hash values).  

Again, this is something where the regular name recognition would still kick in as fallback option if that isn't done.  

 

4) Moderation:  

...people also make like .pbp files out of their PlayStation images and all kinds of compressed files that aren't gonna be listed in any .dat file.  
I would expect a lot of hash values being added to the database with NO idea where they even came from, which makes it hard to impossible to moderate and keep the DB clean after the fact.  
It might be a good idea to have people who add hash values manually give a bit more info on filename and source and have that visible to other moderators.  
Or maybe just stick to the actual clean .dat files so it doesn't end up being a huge mess and just don't allow user contributions. ...but then that wouldn't help at all with our current C64 situation.  ...mmmmh.  

I guess the question is what the goal is of doing this, do we want to promote people using clean and up-to-date .dat sets or do we just want LaunchBox to recognize everything it possibly can, even including the old "GOOD" sets that everyone will tell you not to use anymore because its full of outdated bad dumps.  
That i can't answer for you...

 

...well now you got a few more things to think about.

Edited May 9, 2020 by Z3R0B4NG

[SiriusVI]

9 hours ago, Z3R0B4NG said:

...well now you got a few more things to think about.

Those certainly are a lot of things to think about. Here are some of my thoughts:

1. Just going by a file's MD5 hash would only be possible for certain closed sets of roms. Couldn't these sets be separated from different roms? Launchbox already has a MAME full set importer, so why not add an importer for, e.g. No-Intro sets, such as:

• "Import No-Intro Set"
• Import full set
• Import one (or more) specific systems
• Region / Version priorities, and so on.

This way, people who have a specific set can import that set. people who have custum sets, just import as usual.

2. Redump and other sets for CD-Systems might be a bit problematic for people like me. I tend to convert all bin/cue roms I have to .chd whenever possible. Depending on what set people use as a basis for converting to chd, different MD5 checksums will be created. But even if people stick to e.g. redump as their basis for converting their games to chd, someone would have to collect all the checksums of these converted games...

3. Computer Systems

These are still the cause of lot of headache for me. There doesn't seem to be a clear choice for me on which set to use. For consoles and handhelds it's No-Intro, no question. For CD-Based Systems it's Redump. But for computer systems, I really don't know. Which set would be the basis for a MD5 hash check?

• There is Mame Software List roms which come in a neat package and are updated regularly, but unfortunatley, games, applications and test programs are all mixed together. ALso roms with different file extensions are grouped together, but sometimes roms with different file extensions need different treatment in order to get them to run with MAME.
• TOSEC has a its sets neatly organized into games, applications, educational, and so on. Plus it separates roms by different file extension. However, TOSEC seems to have a lot of duplicate roms (with different file extensions) and therefore overall fewer roms than mame at least for some systems. In some cases TOSEC misses a whole categorie of roms compared to Mame. E.g. Mame has Apple II cassette tape roms, whereas TOSEC doesn't seem to have them. Also, I've realized that some TOSEC roms wouldn't run properly whereas the MAME SWL equivalent would, so maybe MAME's dumps are of better quality.
• There is also the fact that TOSEC has many games that MAME SWL does not have and vice versa.

So how would we decide which set is to be used?

 

If you ask me, as a start Launchbox should add the option to import complete No-Intro sets through a separate importer while still giving the option to import roms from other collections.

[CriticalCid]

10 hours ago, Z3R0B4NG said:

I would suggest, just like the alternate names field to let us add hash values to database entries.  
Instead of regions you could have a drop down menu with No-Intro / Redump / TOSEC / GOOD / other.  
There are roms that of course do not come from the common sources or are scene rips (C64 scene for example, the stuff used in C64 Dreams).  
 

Another thing i recently learned is that Dat-o-Matic has Parent/Clone sets for No-Intro, those exist so you can make "1G1R" (1 game 1 release) sets with it, with that data i think it could be possible to make it a bit easier because you should be able to extract every hash for a specific game and group them together right from the .dat file.  

The filenames should match for the largest part with the LBDB already, if you can apply the same algorithms when importing roms into launchbox to add these hash values to DB entries then a good chunk of it should happen automagically, anything that doesn't get matched you could put in a list and post it so we moderators can edit it in. 

I completely agree with that.

 

42 minutes ago, SiriusVI said:

If you ask me, as a start Launchbox should add the option to import complete No-Intro sets through a separate importer while still giving the option to import roms from other collections.

There are multiple problems I see with this. MAME is only updated once a month and offered in a neat complete package so it’s easy to keep track of your ROM versions. No-Intro, Redump, etc. on the other hand update their sets daily (sometimes even multiple times a day) and on top of that has every system its own .dat file. You would need the exact ROMset based on the .dat versions Jason includes in the importer which isn’t easy to get for the average LaunchBox user as you won’t find full No-Intro sets for every .dat version out there. You also can’t expect the users to rebuild their ROMs with tools like CLRmamepro, ROMcenter, etc. to make it match with the one from LaunchBox. Same thing applies if the users provide their own .dat files because most full set downloads don’t have the .dat files included and then you are back at having the users to need to rebuild their sets by themselves.

While I personally would be really thankful for this feature and have no problem rebuilding my sets, I know that Jason is very hesitant when it comes to spending a lot of development time on features that will only be useful for handful of power users.

This also wouldn’t solve the issue of associating the games 100% correctly with the LaunchBox database entries.

[CriticalCid]

My own thoughts on this topic:

I would love to see proper .dat file support in LaunchBox. I‘ve been wanting this feature for so long that I already have giving up that you will ever add it to be honest.

The first thing you should definitely add is support for hash values in the database as the file names in these sets change from time to time (that‘s especially the case for Japanese games because there’s so many ways you can translate the title to English). And of course, add a hash check in the ROM importer to make use of this data.

 

No-Intro and MAME are afaik the only ROMsets which offer proper support for parent-clone relations. You can get the latest parent-clone .dat files from No-Intro in a single package from here:

https://datomatic.no-intro.org/?page=download&op=daily

Just choose „P/C XML“ in the dropdown menu and download it. It gets automatically updated every 24 hours with the latest .dat versions at that time.

 

When you merge the data with the LB database you have to be extremely careful as the LaunchBox importer already matches some ROMs wrong. Also there are some cases where two different games have the exact same title. Casper for the SNES as example. The western and japanese versions are entirely different. Here you can make use of the parent-clone relations in the No-Intro .dats as they also treat these 2 versions as different games.

https://gamesdb.launchbox-app.com/games/details/2681

https://gamesdb.launchbox-app.com/games/details/112207

Or just look at this thread for other examples

 

I also would highly suggest to also add the file names as alternative names (in case they don’t already exist). That’s especially useful for disc based systems, as @SiriusVI has already mentioned, some people (myself included) convert them into more storage-saving file formats like .chd. In this case the hash values from the official .dat files would be useless but the file names on the hand should still be really valuable.

You can get Redump .dat files from here:

http://redump.org/downloads/

[SiriusVI]

9 minutes ago, CriticalCid said:

No-Intro, Redump, etc. on the other hand update their sets daily (sometimes even multiple times a day) and on top of that has every system its own .dat file.

Didn't know that. In that case, scratch what I suggested ??

[Z3R0B4NG]

12 minutes ago, SiriusVI said:

why not add an importer for, e.g. No-Intro sets

hmm... MAME updates once each month, i don't know how often LaunchBox refreshes its data for that, but No-Intro .dat files get compiled on a daily basis at Dat-o-Matic (at least the daily pack here https://datomatic.no-intro.org/?page=download&op=daily ), but people may have roms from 6 months ago or just random single roms from wherever instead, i would not want to split that into different import processes, many people will not know what they have.  

You have to also consider the Userbase will have all levels of know how, from hardcore collectors to absolute noob level casuals who never heard about No-Intro or dat files or any of that stuff and just got some roms from a friend on a disc and want to "throw it in there" etc. and ideally you want LaunchBox to match stuff correctly, no matter who imports roms or where they came from.  
The hash values from the known dat files would really just add another, more reliable level of accuracy to the existing matching process.  

And AFAIK LaunchBox, when importing roms only scans for database entries of that specific System you told it to import, so hash values of say SNES and Sega 32X would have no chance to ever collide because it isn't even looking there, not to mention that hash values are unique and should NEVER collide for different files anyway, heck that stuff is precise enough for copyright lawyers to sue torrent users over, hash values are practically a digital finger print, i don't think it is even possible for 2 different files to randomly have the same values ...not AFAIK anyway.  

 

I am personally not using TOSEC, not yet... but from what i've seen they upload full packs only every once in a while and post them in news posts, which is a weird contrast to how automated Dat-o-matic does it for No-Intro.  

Ideally you want to support all hash values from all .dats and databases you can possibly get your hands on.  

But i'm just not sure if i would trust users with submitting hash values, if they submit it to a wrong game or system, there is no way to know that it was an error after the fact, because all you got is a bunch of numbers and letters that human moderators can't understand unless they randomly happen to have the identical file with the identical hash value and notice a wrong match has happened somehow.  

[Jason Carr]

Thanks for all the feedback, guys. To be completely honest, I'm not 100% sold on integrating DATs directly into the LaunchBox app itself, but that would come later anyways. Step one would be integrating No-Intro data into the games database, and using that on imports to improve ROM recognition. It can also be used to improve the data on the DB. It'll take a lot of manual work to match everything up, but I can see it being a big improvement. I don't think we can trust end-users to populate the games database with No-Intro data, so that would have to be done by us.

[kurzih]

I agree with a lot of suggestion here! For me I would be already happy for just a solution for this case @Jason Carr:

Ideally games with identical names in the same platform (but are not the same game) could all be found from the database and all images would not get mixed up <= this is the biggest issue, media getting mixed up (since after downloaded, the images take their name from the default title name in our images folders). And not forgetting about EmuMovies manuals and videos, where it gets even more complicated if there are titles named the same.

So far this has been a workaround fixing it within the database:

"Proper title name (Publisher)" or if not possible

"Proper title name (Year)"

+ ALT names as "Proper title name" and all variations to make it possible for the search function to find it.

So this would name the title with a unique name, but still keeping the proper name it should have according to the database rules. Might be ugly looking, but at least it works for now.


Here's a good example of the problem, as long as no one has fixed it yet, until you read this:

1. Choose the Commodore 64 platform and press "Add" and search for: "Battleship" in Launchbox App

2. It should give only one result:  Battleship (1/1/1988)
ID: https://gamesdb.launchbox-app.com/games/details/88189 - not the most well known battleship game for C64 I might add.

In reality there are this amount of games called Battleship for Commodore 64 in the database, but only one can be found in the app, because they all have the same main name:
https://gamesdb.launchbox-app.com/games/details/88185
https://gamesdb.launchbox-app.com/games/details/88186
https://gamesdb.launchbox-app.com/games/details/88187
https://gamesdb.launchbox-app.com/games/details/88188
https://gamesdb.launchbox-app.com/games/details/88189
https://gamesdb.launchbox-app.com/games/details/113365
https://gamesdb.launchbox-app.com/games/details/32326 (this is the one I would have wanted to add, from Elite/Epyx etc.)

Adding unique names (as the workaround above) to those, like "Battleship (Dynastar)" and as alt name "Battleship", would make the app find that entry when searching for "Battleship", and also download images/media that will not get mixed with the other titles called battleship.

If a more technical/program based solution is found with .dats/hashes or something else and we could do without that "Game name (Publisher)" solution, that would be superb! Though I don't really mind having titles named like that as long as they are found in the app and the images are correctly united with their games.

Emumovies videos & manuals are another topic since they tend to behave even more problematically with titles named the same. The workaround so far for that is quite manual and on the user's side (making a copy of one video and downloading again for the other title(s) and then renaming the copy back to the one that was replaced/overwritten.. something like that :D).

EDIT: updated entry to reflect the latest development: The Main Default name still needs to be unique on the same platform, but the Alternative name cannot be identical anymore/cannot be used as it was since it will not show up if there is multiple identical alternative names.

Edited December 26, 2020 by kurzih