MAME Software Lists Import Tool - plugin

[JoeViking245]

How are you bezel creation skills?  

[mcfilmmakers]

On 5/29/2020 at 8:52 PM, JoeViking245 said:

How are you bezel creation skills?  

lol.

So now that 221 is out... what is the update process supposed to be after i've gone through the clrmamepro process? Dragging the files into launchbox results in Titles matching the rom names... which is garbage. The intention is to merge new roms with same names into additional apps of already existing games. The plugin can import the entire platform from the hashfile but I don't know how it behaves with updated sets...

I'm afraid to move forward so I'm asking and suggesting at the same time...

First situation:
Am I correct in assuming that setting my roms source folder to my MAME - Update Software List ROMs (v0.220 to v0.221/platformIwant) folder and the copy my roms to destination towards the location of my software roms folder (launchbox/Emulators/MAME/software/platformIwant) then the plugin will import only the updated roms instead of the entire list for the platform?  UPDATE: Answer: no.

UPDATE: I ran a quick test with a smaller platform that had some changes: Milton Bradley MicroVision. I tried the above and the following happened:
The platform game list went from 15 to 21. However, 221 only added 3 games (1 is a clone). As such, the expected game lsit should have gone from 15 to 17. Here's the rundown of what happened:
   - the game Blockbuster saw it's name change to Block Buster. Since the name changed, it was imported as a new title in Launchbox even though the filename hasn't changed. Suggestion: the plugin should check the filenames for potential same game matches.
   - the game Connect Four had 2 clones in 220 (connect4, connect4o, connect4t). When importing 220 (combine clones), the plugin preferred the Connect Four (8021) title. In 221, connect4t was removed. The plugin imported connect4 and connect4o as a new entry, preferring the title of Connect Four (TMS1100 version) base on the connect4.zip file... I don't know if this is because the name of connect4.zip in 220 was different. I'm going to assume that is the case since connect4t was removed. As such, the suggested solution is the same as above.
   - the game Star Trek Phaser Strike saw it's name change to Star Trek: Phaser Strike. See above suggestion.
   - I had deleted the game called Demo when I imported 220. Importing 221 brought that game back. Complex suggestion: It would require creating a new file for this to work, but the ability to create a black list per platform would be awesome to prevent this. Especially with larger platforms where I deleted in some cases HUNDREDS of magazine files, creating a blacklist on the Select a Computer/Console to import screen by selecting a platform, then highlighting the game you want to black list (ability to multi-select with shift/ctrl more than one at a time an added bonus) and then pressing on a blacklist game button would avoid importing those games the user doesn't want. A whitelist button would allow the user to undo their blacklist selections.
   - The two clones combined on import as expected, however both clones are new so I couldn't test how this interacts with pre-existing entries with matching titles.

So, in this small set, I had to delete 4 entries. But we have to consider platforms with thousands of entries (such as Apple 2, Amstrad CPC, etc) where it isn't possible to go through the list manually.

Second situation:
Say you already have imported a very large platform (Apple II cracked for example). The set has MANY same game titles with different names (Zork I for example has 6: Zork I (release 26...), Zork I (Release 75...), etc). Naturally the plugin didn't combine these clones because they don't have the same names, nor would I expect it to. So I manually combine them into Zork I which places them as additional apps with one of them acting as the main entry. If I import the platform again, then it will re-import those combined games (since they have different names) and I don't want them to do that... or am I wrong and it will ignore them because they are found in the additional apps of another title?

An easy fix here (if it is a problem), perhaps, would be to have a checkbox on the select how you want your roms imported screen under skip Non Supported games which would say skip all games in Additional Apps with a warning that it can increase importing time (for obvious reasons)... (it should check the file name, not the app title).

CONCLUSION: Checking filenames in addition to game titles will avoid double entries where the title of the game was changed between Mame versions. It won't stop double entries where the game title AND the filename changed, but it'll stop 99% of those situations since it is rare both change at the same time. Allowing the creation of blacklists will also prevent having to delete already deleted roms when they are re-imported. I can see the suggestion to simply delete the zip file from the folder coming but what if you want to keep the file in case you might want to import it in the future? (for example, I want to blacklist a working game that isn't 100% emulated to my liking but in the future it is updated to a state I want). Deleting the file means I'll never see it again since it won't be placed in the update packs unless the rom itself changed).

Also, another suggestion: is it possible to have the plugin auto select MAME as the emulator on the choose an emulator screen (as long as an emulator with that name exists)... just another QOL thing that reduces those clicks....

and another: the option to remove all text in brackets from game titles (nobody wants to see Connect Four (TMS1100 version) in the title, they just want to see Connect Four.... should accommodate for multiple instances of brackets (ie> Connect Four (TMS1100 version) (clean cracked))

I ask and suggest because I love. Your work has already made the unimaginable possible as is. I would help if I knew how...

Thank you.

Edited June 1, 2020 by mcfilmmakers

[JoeViking245]

Dude!!!!  It's designed to import!  Not update!    Now that I've got that out of my system....

Here's how it's setup...

"Copy AND Import" - the Copy roms part:
A - It will copy AND overwrite (replace existing) all roms as per your selected criteria.  All.  No clones.  No prototypes.  No non-supported games.  Or a viable combination there of.

B - It does not and will not look to see if an existing rom no longer exists (is obsolete) in the current [updated] set.

Theory:  
Part A - Say a game was updated but the rom's name didn't change.  You'd want the original overwritten (replaced) by the new one.  
Part B - Don't want to delete something that is 'obsolete' per MAME, but the end user wanted to keep.

"Copy AND Import" / "Import Only" - the Import part:

Importing separately - If the Game Title does not already exist in the platform you're importing to, import it.

Importing additional Apps (Clones) - If the clones' parent already exists in the platform you're importing to, AND the clone is not already an additional app to that parent, import it.

 

Your First Situation.  Hmmm.... I think "Yes."  If the file to copy doesn't exist in the Source folder, it skips it and jumps to the next one "in the list".  So if your Source folder is your Update Pack folder, I suppose the Copy process would actually go a lot faster.   Nice one! I like it.  Hmmm... unless you do the clrmamepro process 1st. ?

Second Situation.   Unfortunately any subsequent changes you make to your LaunchBox Platform in this manner will not be addressed with a new (eh hem.... update) Import.  The process to scan all existing Parents and all existing Additional Apps associated with each Parent to see if a game to be imported is listed as a Parent or an Additional app would, in-the-end, utilize quite a bit of resources and will definitely increase Import time.  Ain't happening.  Easy fix 'taint so easy.  

NOW.... if the games in question were not in your update pack Source folder, I   t h i n k  you'd be OK.  One sure way to find out. lol   Plan B, edit the hash file before importing and note those files as clones.  Example:

change this
<software name="cpm8607">
  
to
<software name="cpm8607" cloneof="cpm86">

When updating (as much as can be) be sure to update your hash files as well for Importing.  (Though "common knowledge", I try not to take things for granted)

2 hours ago, mcfilmmakers said:

is it possible to have the plugin auto select MAME as the emulator

Yes.  Funny thing is I thought I had done this.  But I guess if you only have a couple Emulators and the 1st one on the list is "MAME", it would appear that way. lol  I've added this to my to do list.

As Always, thank you for your love and support.  Keep the comments, concerns and suggestions coming.

[mcfilmmakers]

32 minutes ago, JoeViking245 said:

Your First Situation.  Hmmm.... I think "Yes."  If the file to copy doesn't exist in the Source folder, it skips it and jumps to the next one "in the list".  So if your Source folder is your Update Pack folder, I suppose the Copy process would actually go a lot faster.   Nice one! I like it.  Hmmm... unless you do the clrmamepro process 1st. ?

Second Situation.   Unfortunately any subsequent changes you make to your LaunchBox Platform in this manner will not be addressed with a new (eh hem.... update) Import.  The process to scan all existing Parents and all existing Additional Apps associated with each Parent to see if a game to be imported is listed as a Parent or an Additional app would, in-the-end, utilize quite a bit of resources and will definitely increase Import time.  Ain't happening.  Easy fix 'taint so easy.  

NOW.... if the games in question were not in your update pack Source folder, I   t h i n k  you'd be OK.  One sure way to find out. lol   Plan B, edit the hash file before importing and note those files as clones.  Example:

change this
<software name="cpm8607">
  
to
<software name="cpm8607" cloneof="cpm86">

When updating (as much as can be) be sure to update your hash files as well for Importing.  (Though "common knowledge", I try not to take things for granted)

Yes.  Funny thing is I thought I had done this.  But I guess if you only have a couple Emulators and the 1st one on the list is "MAME", it would appear that way. lol  I've added this to my to do list.

As Always, thank you for your love and support.  Keep the comments, concerns and suggestions coming.

Yeah, I updated via clrmamepro first. I'm just thinking of how the plugin grows from here and the natural path in my estimation is to enable updating.

I ask because my source folder literally had 3 zip files in it. My platform before import had 15 titles. My destination folder had 18 files (I already did the update with clrmamepro so it was at 15 before the update: 15+3 = 18). After importing, launchbox had 21 titles (I expected to see 17 (I had manually removed 1 entry so 15 + 2 (remember, 1 of the new games is also a clone) =17)). Which means it didn't skip what wasn't in the source folder because if it did, it should have only added the 3 files resulting in 2 new titles. The culprit, as demonstrated, is the sole reliance on the game title during import which doesn't account for MAME changing the name of that title.

As such instead of:

"Copy AND Import" - the Copy roms part:
A - It will copy AND overwrite (replace existing) all roms as per your selected criteria.  All.  No clones.  No prototypes.  No non-supported games.  Or a viable combination there of.

B - It does not and will not look to see if an existing rom no longer exists (is obsolete) in the current [updated] set.

"Copy AND Import" / "Import Only" - the Import part:

Importing separately - If the Game Title does not already exist in the platform you're importing to, import it.

Importing additional Apps (Clones) - If the clones' parent already exists in the platform you're importing to, AND the clone is not already an additional app to that parent, import it.

 

It COULD be:

"Copy AND Import" - the Copy roms part:
A - Create a list of files where the filename in both the source and destination folders matches (aka find all same games based on filename). Let's call this GameMatches.txt...

B - It will copy AND overwrite (replace existing) all roms as per your selected criteria.  All.  No clones.  No prototypes.  No non-supported games.  Or a viable combination there of.

C - It does not and will not look to see if an existing rom no longer exists (is obsolete) in the current [updated] set. <- that's cool. I don't expect the plugin to do this. 

"Copy AND Import" / "Import Only" - the Import part:

Importing separately - If the Game Title does not already exist in the platform you're importing to, check if the filename.zip in the destination folder (or source folder... we could use either one) exists in GameMatches.txt. If it doesn't, import it (this is a new game); if it does exist, do not import it (the new game title is different from the old game title but they are the same filename.zip therefore, they are the same game) or, go a step further... prompt the user if they want to import anyway. We can show a prompt saying "Blockbuster and Block Buster share the same filename, would you like to import the game as a new title, rename the already existing title (edit the launchbox platform xml of the title to match the software list title), or skip importing this title?" with the appropriate options.

Importing additional Apps (Clones) - If the clones' parent already exists in the platform you're importing to, AND the clone is not already an additional app to that parent, import it.

As is, the current solution to the problem is to delete the platform entirely and re-import it entirely. Implementing the above would save a large amount of time and effort for the user. Obviously, it may be beyond the scope of your intentions for the plugin and that's fine.

 

I wouldn't expect the second situation to be implemented, but it is a dream scenario. The problem is that when you have literally hundreds of files you manually deleted, it simply isn't possible to edit the hash files manually every time you update MAME (once a month), hence the blacklist. The alternative here would be to keep the title in Launchbox and set it to hide. Not the end of the world, it's just an OCD thing.

[JoeViking245]

Ya, 'updating' is beyond the original intent of the plugin.  But is the next logical progression for a version 4.0.  I'll add to my "I'll think about it" list.

I'm curious to look into your 3 - 15 -18 - 21 - 17 files/games example.  I guess it's more of a morbid curiosity.  Which sofltlist set was that in?

[mcfilmmakers]

4 hours ago, JoeViking245 said:

Ya, 'updating' is beyond the original intent of the plugin.  But is the next logical progression for a version 4.0.  I'll add to my "I'll think about it" list.

I'm curious to look into your 3 - 15 -18 - 21 - 17 files/games example.  I guess it's more of a morbid curiosity.  Which sofltlist set was that in?

Milton Bradley Microvision... microvision.xml

[JoeViking245]

OK.  That one is all messed up.  lol  But I think I see how you ended up with 21.  The Game.Title (description) was the biggest killer (as you mentioned).  And the Connect 4 series didn't help matters either...

                old                                                               new
<software name="connect4">                                  <software name="connect4o" cloneof="connect4">
<description>Connect Four (8021)</description>              <description>Connect Four (8021 version)</description>
sha1="971f9117dc809f7b9796134208b47ea34990ae71"/>           sha1="971f9117dc809f7b9796134208b47ea34990ae71"/>
  
<software name="connect4t" cloneof="connect4">              <software name="connect4">
<description>Connect Four (TMS1100)</description>           <description>Connect Four (TMS1100 version)</description>
sha1="e93d848784205ab49234ad26718e04d267b776ed"/>           sha1="e93d848784205ab49234ad26718e04d267b776ed"/>

As you can see by the hash #'s, the Parent became the Clone, and the Clone became the Parent.  So if someone merely copied the update pack into their fullset without running crlmamepro 1st, they wouldn't have connect4o and their connect4 wouldn't be the actual parent.   connect4t wouldn't be seen and I guessing none of it would work in MAME.

So I am actually looking into the logistics of fitting in Update scenarios.  Not sure how pretty or not it's going to be (the code OR the outcome).  We'll see.  

[wallmachine]

On 6/3/2020 at 8:37 AM, JoeViking245 said:

So I am actually looking into the logistics of fitting in Update scenarios.  Not sure how pretty or not it's going to be (the code OR the outcome).  We'll see.  

You go dude! Maybe when you figure this out and get the time an update feature for MAME non-softwarelist as LaunchBox doesn't do this either.

Would this software lists import work for 'non-arcade' games in the MAME (non-merged) set too?

 

Edited June 4, 2020 by wallmachine

[JoeViking245]

Thanks.  But I don't think this is going to happen.  Not by me at least.  Too many little nuances to do it properly, but I have found some checks and balances I can add in that will [hopefully] help.

- If they're simply adding new roms, the plugin already covers that.
- If they change the Game.Title, you'll end up with 2 Titles pointing to one rom. (Blockbuster, Block Buster) Delete the old one.
- If they delete a rom altogether, manually delete it from your Platform (and roms folder).  Or not.
- If they rename a rom, my proposed checks and balances may cover that. May need to delete something as noted above.
- If they rename a rom AND rename the title AND re-associate a rom AND add a rom (see "Connect 4" example in previous post), my proposed checks and balances may cover some of that.  May need to delete something(s) as noted above. but I make no promises.

After adding in the 'checks and balances' (and assuming they work),  I may add in an audit feature that will display differences between the hash.xml and LB's Platform.xml mainly covering discrepancies between Titles and rom names.  Again, no promises.

Worse case scenario, delete the Platform and import anew.   You'll need to re-import the metadata, but not the media (assuming no changes in the Game.Title).

[JoeViking245]

*New v3.0
- New Audit Feature.  Compare your existing softlist Platform against the MAME hash.xml.
    Will cross check Title and Rom names.
        - Select the respective hash.xml and Platform
        - Click Audit (last page)
        - Any differences along with a brief description will be shown.
        - Option to select files (if any) and have them removed from your LaunchBox Platform.
        - Will check main Titles as well as Additional Apps (if you imported them separately).
    This does NOT check rom integrity. That's between you and Clrmamepro (or related utility).

[isamu]

Hi Joe and thanks for your continued work on this amazing plugin. I have a question......

 

I'm trying to get my PC-Engine CD and Sega CD games to work. They are all in separate folders and in CHD format. None of them are archived. How do I get launchbox to scan and import these games? I chose the option to scan them as CHD files(second option) rather than zip files(MAME option). Is this the correct method?

[JoeViking245]

@isamu Thank you very much for the kind words!   Questions, thoughts, input, opinions and suggestions are always welcome.

The 2 options are for how it sets up (Imports) the rom file location into LaunchBox.  So depends on how you plan to emulate them.

If you are planning to (attempt) to run softlist chd's with MAME as your emulator, definitely choose the 1st option.  MAME launches the games using the [short] zip filename (just like Arcade), then (because it's smart) it knows to look for the chd to actually load and play.  

For Other emulators that are able to run the MAME chd's, they need to know the full chd file name.  (A request was made during early development to add this option [full_file_name.chd v.s short_MAME_name.zip])

Both options will give the proper Title so LaunchBox can scan for metadata (the original intent of this plugin)

 

1st option "MAME"

 

2nd Option "Other"

[DOS76]

PC-Engine CD (and TG-CD) work differently and need additional commands then most typical systems if I remember correctly so it may not actually work as easily in LB as some of the other systems.

[DOS76]

I believe it needs something along the lines of the below syntax but unless there is a variable I don't know you would need to add the command to each game individually with the proper game name to get it to work
pce -cart scdsys -cdrom GAMENAME

[JoeViking245]

Not tested, but my notes do mention the "Super CD_ROM System..." (scdsys).  So this all makes sense now.  But you wouldn't need to add it to each game individually.  In your MAME emulator under Associated Platforms for "NEC PC-Engine / Turbografx 16 CD-ROMs" (or whatever name you gave it),  set the Default Command-Line Parameters to...

pce -cart scdsys -cdrom

.. and you're good to go.Thanks @DOS76  

The Atari 2600 cassettes is similar in that it needs the Arcadia (Starpath) "SuperCharger" loaded as well.

[sundogak]

@JoeViking245 Using the beta plugin with LB 11.3 beta 5, I notice that if you run the audit platform within LB, that anything where the plugin was used to import it has these added fields in the end of the audit grid.  "Alternate, Clone of, Casino Game, Mature, and Runs in Mame" are getting added. None of my non-SL import platforms have these fields. One was an existing platform (Apple IIGS) and one was a newly imported SL I did for Atari 800. Seems to be partial of a MAME column setup (second snapshot). 

SL Platform using Import Plugin:

Normal MAME view using built in LB fullset import:

Not a huge issue since just info window, but interesting in that would think would go one way or other (i.e., none of MAME fields or all of them).

[neil9000]

16 minutes ago, sundogak said:

@JoeViking245 Using the beta plugin with LB 11.3 beta 5, I notice that if you run the audit platform within LB, that anything where the plugin was used to import it has these added fields in the end of the audit grid.  "Alternate, Clone of, Casino Game, Mature, and Runs in Mame" are getting added. None of my non-SL import platforms have these fields. One was an existing platform (Apple IIGS) and one was a newly imported SL I did for Atari 800. Seems to be partial of a MAME column setup (second snapshot). 

SL Platform using Import Plugin:

Normal MAME view using built in LB fullset import:

Not a huge issue since just info window, but interesting in that would think would go one way or other (i.e., none of MAME fields or all of them).

Thats a newly added feature. I have them as well and i have never used this plugin.

This is my standard MAME import imported via Launchbox's fullset importer.

[sundogak]

Appears to be an inconsistency in how LB is doing things in the beta.  This is the view I have in MAME which didn't use the plugin (more fields than you have).  Seems like all these fields would be common to both your view and to the others where this plugin was used (i.e., anything where MAME importer process used either via plugin or built in).  Maybe a minor bug in the beta then.

Edited August 2, 2020 by sundogak

[JoeViking245]

@sundogak  It wasn't me! I swear!  lol   (thanks for having my back @neil9000  )

Seriously though, for clarification of what actually this softlist importer plugin does add to LaunchBox...

A new Platform (named per your choosing)
   A new 'game' (Title/Name per MAME's hash file)
   The 'ApplicationPath' to that new 'game'
   The 'DateAdedd' (DateTime.Now)
   The 'DateModified' (DateTime.Now)
   The associated Emulator [Id] (per your choosing [i.e. MAME])
   And checks the box "Broken" (if you imported an "unsupported" [per the hash file] 'game' )
For the selected Emulator:
   Add's a new Platform
   Add's the command line for the new Platform (to the best of my ability with the given [hash file] information)
   And checks the "Default" box

The rest of 'what you see is what you get' is all LaunchBox.

I suppose I could force a "check" to the "Non-Arcade" field.  Though I'm not sure if it would 'stick' after importing metadata.  In fact I don't know if "Broken" sticks or not.  Hmmm....  

(Out of morbid curiosity) I wonder what your audit would look like before you add metadata and media (to the new softlist Platform).  Obviously you won't have any images, videos or manuals.  But are the extra boxes there?  Also, if you update you metadata/media in Arcade, will those "new" boxes now show?

[wallmachine]

Item 26!