Duplicate MAME games when importing and crashing with "tablepop".

[Phosky]

Hello,

I'm having problems with the latest version of LaunchBox, a version that I installed fresh.

First, when I imported the games through the "MAME Arcade Full Set" option I selected the "Skip tabletop games" option, but despite that, It imported chess games, which LaunchBox cannot identify (for example, Addometer Calculator and Chess Companion).

Anyway, the biggest of my problems is something else. When I added the games the way I told you (Import>MAME Arcade Full Set), I received a total of 3078 games (full ROMSET 0.262) with my selection of preferences when importing the games.

The next thing was to select the "Scan for removed roms" option. I received a notice that 1826 ROMS were being removed. I hit OK. Next, I hit "Scan for added roms" (NOT from Import>ROM Files) and 2906 new ROMS appeared. After accepting, right now I have 5829, and ALL of them are duplicates, as you can see in the screenshot. I know that if you choose the "Import ROM files" option and select "Force importing duplicate games" the ROMS will be duplicated. But that has not happened. I have only scanned for added roms.

As you can see in the following screenshots, it seems that it considers the ROMS as if they were different, despite being exactly the same and being in the same folder.

Thanks.

 

EDIT

I've started again installing flesh LaunchBox, trying to replicate the problem.

If I import all the ROMs using the "MAME Arcade Full Set" option, It shows me 3078 games. Now, If I choose "Scan for added roms", it show ZERO (0) roms. Now, If I choose "Scan for removed roms", it shows 1826 roms to delete (why? These roms exists in the correct folder). I choose DELETE, but now, I've 3000 games (3078-1826 is not 3000...). Now, I scan for added roms: 0 new roms.

 

EDIT 2

I've selected ALL ROMS and deleted all them (Displaying 0 of total 0 games). Tools>Scan for added roms. "No additional roms were found"..............

 

EDIT 3

Now, I've added ALL roms using the "MAME Arcade Full Set" option, without any Skips. In the preview I see again 3078 games (impossible, it is a bug), but at the end of the process, the real number is 19825. OK. I choose again "Scan for added roms", it show ZERO (0) roms. Now, I choose "Scan for removed roms", it shows 11489 roms to delete. After removing, I can see at the top: Displaying 14376 games.

 

I'm going to test it with Nintendo 64 ROMS...

Edited February 2 by Phosky

[JoeViking245]

2 hours ago, Phosky said:

It imported chess games, which LaunchBox cannot identify (for example, Addometer Calculator and Chess Companion).

These 2 particular games are 'relatively new' additions to MAME and have not yet been categorized into LaunchBox's Metadata.  In fact, don't see them in the database either.

 

2 hours ago, Phosky said:

The next thing was to select the "Scan for removed roms" option. I received a notice that 1826 ROMS were being removed.

When using the MAME Full Set Import Wizard, LaunchBox doesn't really even look at your physical ROMs (other than to count them to see that you have a [near] full set).  It uses your rmame.exe to create a list of ROMs that version includes, then imports from it, based on your filters.  If your set is a merged or split set, scanning for removed ROMs is going to notice that all clone ROMs are missing.

 

2 hours ago, Phosky said:

Next, I hit "Scan for added roms" (NOT from Import>ROM Files) and 2906 new ROMS appeared.

As mentioned, the Full Set Import Wizard works differently than other [regular] import wizards.

Regarding duplicates being created after scanning for added ROMs, I'm not entirely sure what all is involved in that process (as far as what exactly is cross-checked. Source may very well be one of the checks.).  That said, because of the uniqueness of MAME, it's not advised to use that feature for MAME ROMs. 

 

Your 1st "Information" screen shot game came from the Full Set Import in which the Source came directly from the 'list' created by mame.exe.  The 2nd one is most likely from the Scan for Added ROMs, where the Source is truncated, and I presume came from the LaunchBox database.

 

Again, because of the uniqueness of MAME and its ROMs, scanning for added or removed ROMs for the "Arcade" platform is not advised (for reasons in which you've discovered).  For adding ROMs (i.e. with a MAME updated release), rerun the Full Set Import Wizard.  For removed ROMs, it's not very often that MAME will remove games with its updates.  And with over 3000 games in your Arcade platform, you probably won't even notice.   But if you do, select the game in LaunchBox and delete it there.

[Phosky]

Thanks for the quick reply.

MAME releases updates every month that add ROMS, but also remove them, or rename them. Between the last version and the second to last, after deleting/moving files, about 90 new ROMS were required to be downloaded. Most were adjustments to existing ROMS.

That is, if I understand you correctly, if I download the "full" version instead of the "merged" version, would these problems disappear or would they be minimized?

Edited February 2 by Phosky

[JoeViking245]

16 minutes ago, Phosky said:

Thanks for the quick reply.

MAME releases updates every month that add ROMS, but also remove them, or rename them. Between the last version and the second to last, after deleting/moving files, about 90 new ROMS were required to be downloaded. Most were adjustments to existing ROMS.

That is, if I understand you correctly, if I download the "full" version instead of the "merged" version, would these problems disappear or would they be minimized?

You're welcome.

If you have a non-merged full set (vs. merged or split), these side-effects of MAME will not disappear.  Best case-in-point example is scanning for added ROMs where the Source values are different because of where/how they are respectively obtained.

 

How I handle the MAME updates is, after getting the update pack (156 ROMS in .261 to .262) and running them through ClrMamePro, I'll read through "whatsnew.txt".  If there are new games I feel I 'must have' in my LB collection, I'll manually add them (which doesn't happen super often).   For any removed ROMs, if they're must haves, I move the ROM to a 'Removed ROMs' folder and make note of the MAME version in which it last worked. (I only have one or 2 of those.)  I ignore renamed ROMs because most the time they came across a different variant of it and rename the old one (game.zip --> gameA.zip) and name the 'new' one "game.zip".  No harm. No foul. No loss.  Games they rename without having a new one to replace are probably not in my collection anyway.

 

4 hours ago, Phosky said:

I'm going to test it with Nintendo 64 ROMS...

You should have no issues. 

[Phosky]

23 hours ago, JoeViking245 said:

You're welcome.

If you have a non-merged full set (vs. merged or split), these side-effects of MAME will not disappear.  Best case-in-point example is scanning for added ROMs where the Source values are different because of where/how they are respectively obtained.

 

How I handle the MAME updates is, after getting the update pack (156 ROMS in .261 to .262) and running them through ClrMamePro, I'll read through "whatsnew.txt".  If there are new games I feel I 'must have' in my LB collection, I'll manually add them (which doesn't happen super often).   For any removed ROMs, if they're must haves, I move the ROM to a 'Removed ROMs' folder and make note of the MAME version in which it last worked. (I only have one or 2 of those.)  I ignore renamed ROMs because most the time they came across a different variant of it and rename the old one (game.zip --> gameA.zip) and name the 'new' one "game.zip".  No harm. No foul. No loss.  Games they rename without having a new one to replace are probably not in my collection anyway.

 

You should have no issues. 

 

Well, I've downloaded the non-merged ROMSET from the same site as the merged romset (a 100% reliable website) and the results are much better this time. When I use the "Scan for removed roms" option, now I only get 5 ROMs, when with the merged romset it appeared that there were 1826 extra to remove. I think this is a LaunchBox problem. LB should handle romsets the same whether they are merged, non-merged or split, I believe.

[JoeViking245]

1 hour ago, Phosky said:

Well, I've downloaded the non-merged ROMSET from the same site as the merged romset (a 100% reliable website) and the results are much better this time. When I use the "Scan for removed roms" option, now I only get 5 ROMs, when with the merged romset it appeared that there were 1826 extra to remove. I think this is a LaunchBox problem. LB should handle romsets the same whether they are merged, non-merged or split, I believe.

The results will be better because when scanning for removed ROMs, it looks at the rom path for the game, then looks in the folder to verify if that file actually exists.  With now having the non-merged set, there is a physical [.zip] file for each game entry (for both parent and clone games).  So the comparison is pretty straight forward.

It's quite possible the 5 that were removed are TTL games that don't have physical ROMs as they are built directly into MAME.  PONG is one of those.

With the merged set, all the clones for a particular game are embedded into the parents ROM .zip file.  Your games path to a clone game will point to a "file" that is named for the clone, as MAME sees it.  For example Donkey Kong Jr.  The parent game will point to dkongjr.zip and the clone will point to dkongjrj.zip.  In this case, the clone file doesn't physically exist, so when scanning, that game gets removed.

Conversely when launching a clone game, LB sends the filename only (minus extension) to MAME.  It's MAMEs innate logic that looks for the required files (found inside the zip file(s)) via an order of precedence.  First looking for dkongjrj.zip, then dkongjr.zip.  The 1st physical file it finds, it then looks inside the zip for the required files. And then looking in the 2nd file in the case of a split set.

The beauty of the MAME Full Set Import Wizard is that it will import all games, parents and clones alike, regardless of the set type you have (aka, physical files).

LaunchBox doesn't know if you have a merged, non-merged or split set.  Even if it did, to scan for removed ROMS it would need to know which files are required to be in each sets ROM zip-file, respectively.  Then instead of just looking for the physical file, it would need to go through MAMEs logic (above) plus generate a list of the required files (found inside the zip(s)) and scan the contents of the zips.  Just to verify if the game is a clone, the set-type you have, if the parent exists and has the required files for that clone.  We just increased the time to Scan for Removed ROMs by at least a 100-fold. 

Doesn't make good sense to me.  It makes better sense to just remove the games from LaunchBox rather than removing the physical ROMs and then scan.  Also, removing the physical ROMs in the case of a merged or split set can have drastic results.  If I have a merged set and only play the Japan version of DK Jr., I can't delete the US version of the ROM and still be able to play the other.

[Phosky]

Thank you very much for the extensive and illustrative response, it helped me to understand how everything works. What you say makes sense, and analyzing the content of the .zip would take too much time...

 

Exactly, one of those 5 games is PONG. Curiously, the PONG (which LaunchBox lists, and locates in MAME_ROMS\pong.zip), does not exist in the .dat nor does it appear in the complete non-merged romset folder. I don't understand why LaunchBox adds it. But it solves it later when scanning for deleted roms. There it does detect it and eliminate it.

 

I have enough space on my drive, so I think I'm going to keep the "non-merged" version, because it's the closest thing to any console romset (1 file = 1 game, regardless of whether it's a hack, a clone , and so).

Edited February 3 by Phosky

[JoeViking245]

1 hour ago, Phosky said:

Curiously, the PONG (which LaunchBox lists, and locates in MAME_ROMS\pong.zip), does not exist in the .dat nor does it appear in the complete non-merged romset folder. I don't understand why LaunchBox adds it.

Pong is built into MAME so you will not find a physical file in any ROM set after [somewhere around] revision 0.245.  (no relation to the numbers in my name) But to Launch Pong, you still need to tell MAME what it is you want to launch.  Hence LaunchBox adds a "path to the ROM".  Then of course, LB only sends the file name (without extension) from that path.  "pong"

Just like for clones that don't have a physical file in a merged-set, LaunchBox gets Pong (and 4 other TTL games ) ... well... All MAME games/ROM names from your physical "mame.exe" file.    Using the command "mame.exe -listxml", a list of all the games for that revision of MAME is generated.  From there, LB crosses with the file /LaunchBox/Metadata/Mame.xml to get other data.  Both files are relatively large in size and can take a while to go through.  

Here's an excerpt for Pong from the info generated via -listxml

 <machine name="pong" sourcefile="atari/pong.cpp">
        <description>Pong (Rev E) [TTL]</description>
        <year>1972</year>
        <manufacturer>Atari</manufacturer>

..and from /Metadata/Mame.xml

 <MameFile>
    <FileName>pong</FileName>
    <Name>Pong</Name>
    <Status>good</Status>
    <Developer />
    <Publisher>Atari</Publisher>
    <Year>1972</Year>

 

It's not in the .dat file because they're only [primarily] used for comparing/verifying physical files via their hashes (CRC/SHA1).

 

And that's how ('why') LaunchBox adds it (them).  

 

2 hours ago, Phosky said:

I'm going to keep the "non-merged" version

That's what I use.  Originally because it's way more versatile, but now... because that's what I originally used.  (something about "old habits"?)  I do keep it updated monthly and then add/remove games from LB accordingly after perusing whatsnew.