MAME: how to update it in LaunchBox?

[ci2own]

I have the same problem when importing sets:

I already have all MAME media, and when importing full set, it takes about two hours just checked to look for info and use the roms in his location (not copying or moving).

 

[Jason Carr]

I just did some performance testing with the MAME full-set importer, without downloading any media. Results:

Full set import with metadata, default filters: 2 minutes
Full set import without metadata, default filters: < 1 minute
Full set import with metadata, no filters: 22 minutes
Full set import without metadata, no filters: 2 minutes

It looks like the longer times are just coming from the filters (along with populating the metadata). If you don't filter anything out, you end up with 41,763 games/apps vs. only 6,627 games/apps, so that's a huge difference. Obviously I'm getting much faster times than 2 hours, but that may just be due to the fact that I'm running an i9 9900K. It also could be affected by the size of the existing collection.

[Jason Carr]

13 hours ago, wallmachine said:

How do you convert it into a manually-populated playlist? Is that just to un-check the auto-populate list again?

Yeah, should be that easy.

[Jason Carr]

17 hours ago, wallmachine said:

I believe I narrowed it down for you, see my trials of today: (is this some type of scanning that can't fully utilise HDDs?)

• 10TB HDD - This is the HDD the LaunchBox software and MAME games are located. I then re-imported and it took roughly 2 hours
• 4TB HDD - Moved the LaunchBox software to this HDD and the MAME games stayed on the 10TB HDD, then re-imported and this took roughly 2 hours
• 4TB HDD - Moved the LaunchBox software to this HDD and the MAME games stayed on the 10TB HDD, then re-imported and this took roughly 2 hours
• 4TB HDD - Moved the LaunchBox software to this HDD and the MAME games stayed on the 10TB HDD, then re-imported and this took roughly 2 hours
• 128GB SSD - Moved the LaunchBox software to this SSD and the MAME games stayed on the 10TB HDD, then re-imported and this took roughly 30 seconds

10TB HDD 2 hour import time:

That doesn't seem right, since I can't even get a full set to import in 30 seconds on an SSD and an i9 9900K.

[wallmachine]

3 hours ago, Jason Carr said:

It also could be affected by the size of the existing collection.

I think you nailed it.

That fast speed was due to me using a brandnew install of LaunchBox on the SSD with no platforms/games because I didn't have enough space for my original LaunchBox folder.

For a test I then made some room moved original LaunchBox software folder to SSD, this setup has 60k games (the games stayed on 10TB) and the import took around 2 hours. So the more platform/games I have in the collection the longer it will take to import even without metadata. I don't have anymore systems to import so MAME is the only constant that is always updated, so I deleted an already imported 'small' platform and it too also took some time... how come the size of the collection affects the import speed? Can anything be done about this as its just reading the location of the file?

@ci2own how many platforms/games do you have imported?

Edited May 22, 2020 by wallmachine

[ci2own]

17 minutes ago, wallmachine said:

I think you nailed it.

That fast speed was due to me using a brandnew install of LaunchBox on the SSD because I didn't have enough space for my original LaunchBox folder.

For a test I then made some room moved original LaunchBox software folder to SSD, this setup has 60k games (the games stayed on 10TB) and the import took around 2 hours. So the more platform/games I have in the collection the longer it will take to import even without metadata. I don't have anymore systems to import so MAME is the only constant that is always updated, so I deleted an already imported 'small' platform and it too also took some time... how come the size of the collection affects the import speed? Can anything be done about this as its just reading the location of the file?

@ci2own how many platforms/games do you have imported?

I have about 150 platforms and 55315 games/roms imported.

[legolas119]

On 5/22/2020 at 12:11 AM, legolas119 said:

thanks.

Currently I still not know how to create an auto-populate playlist and then how to convert it into a manually-populated playlist...So, in my case, I have to perform only these 3 steps now:

1) Delete all my MAME games: it means right click on the platform MAME and Delete it

2) Re import using mame full set, and uncheck both "download images from database" and "download media from emumovies"

3) Tool --> Download metadata and media, in order to download media for the added new games.

Usually i update my mame romset every month, but i think that i will not do the same thing in LB too...

I have a question: if I delete the MAME platform, I will loose all the playlist automatically creaded. When I will reimport all the fullset, LB will re create them again, but i think that I will have to rename all the playlist in this way, as I have done the first time. In this way I'm sure that i can take separated the Final Burn Alpha playlist from the MAME playlist. I mean, I named every MAME playlist with "Arcade" prefix, and with "FBA" prefix for Final Burn Alpha playlist. Here an example:

If I delete MAME platform, do you think that I will have to rename all the playlist...again?   I think so...

hi, is it correct what I wrote? Is it correct the way I described to delete the platform? If so, I think that after the new fullset import, I will have to manually create again all the playlist...:( 

thanks!

[ci2own]

On 5/22/2020 at 7:42 PM, Jason Carr said:

I just did some performance testing with the MAME full-set importer, without downloading any media. Results:

Full set import with metadata, default filters: 2 minutes
Full set import without metadata, default filters: < 1 minute
Full set import with metadata, no filters: 22 minutes
Full set import without metadata, no filters: 2 minutes

It looks like the longer times are just coming from the filters (along with populating the metadata). If you don't filter anything out, you end up with 41,763 games/apps vs. only 6,627 games/apps, so that's a huge difference. Obviously I'm getting much faster times than 2 hours, but that may just be due to the fact that I'm running an i9 9900K. It also could be affected by the size of the existing collection.

 

I did a new test with clean LaunchBox setup (No platforms added) and it took me 2 minutes to import Full MAME set with just metadata and default filters (No media/videos).

The same exact import with 150 platforms already set up on LaunchBox took me about 2 hours.

All this tests have been done in an old 3770k processor and standard HD (not SSD). So the conclusions i get with all this tests, are: More platforms imported, more time to import roms. Processor and HD doesn't seem to be too significant for the import speed.

[wallmachine]

Not sure if correlated. I know one is about importing and one is about displaying and one is LaunchBox and the other is BigBox but LaunchBox importing takes a while to import when heaps of platforms/games. Maybe this performance issue is similar to how BigBox lags is slow if you have heaps of platforms/games?

Edited May 25, 2020 by wallmachine

[Jason Carr]

On 5/23/2020 at 8:38 AM, legolas119 said:

hi, is it correct what I wrote? Is it correct the way I described to delete the platform? If so, I think that after the new fullset import, I will have to manually create again all the playlist...:( 

thanks!

You do not have to delete the entire platform, only the games. You can just delete the games by selecting them all and pressing the Del key.

On 5/22/2020 at 2:00 PM, wallmachine said:

I think you nailed it.

That fast speed was due to me using a brandnew install of LaunchBox on the SSD with no platforms/games because I didn't have enough space for my original LaunchBox folder.

For a test I then made some room moved original LaunchBox software folder to SSD, this setup has 60k games (the games stayed on 10TB) and the import took around 2 hours. So the more platform/games I have in the collection the longer it will take to import even without metadata. I don't have anymore systems to import so MAME is the only constant that is always updated, so I deleted an already imported 'small' platform and it too also took some time... how come the size of the collection affects the import speed? Can anything be done about this as its just reading the location of the file?

@ci2own how many platforms/games do you have imported?

The size of the collection affects the import speed because of duplicate/combining checks and other related things. I'm sure there are optimizations that could be made though. I'll keep that in mind; might be able to improve it at some point here.

 

[wallmachine]

6 hours ago, Jason Carr said:

You do not have to delete the entire platform, only the games. You can just delete the games by selecting them all and pressing the Del key.

The size of the collection affects the import speed because of duplicate/combining checks and other related things. I'm sure there are optimizations that could be made though. I'll keep that in mind; might be able to improve it at some point here.

 

just a theory not sure if you already do or not.

Maybe an option when you start to import, treat platform as separate (so it imports without checks, i don't know what other related things you check) so then you don't need to do checks for games at least. I know you have combine roms with similar name (can't remember the name) but i assume this does the check?

[legolas119]

17 hours ago, Jason Carr said:

You do not have to delete the entire platform, only the games. You can just delete the games by selecting them all and pressing the Del key.

The size of the collection affects the import speed because of duplicate/combining checks and other related things. I'm sure there are optimizations that could be made though. I'll keep that in mind; might be able to improve it at some point here.

 

thanks, I deleted only all the games, but the platform "MAME" disappeared and its Playlist were still present, but empty and without a parent. I imported again the fullset without media: 2942 games (instead of 2945 so i supposed that 3 romes were removed in mame 0.221).  Now I have to manually associate every plalylist to "MAME" as parent...  I'm doing it, but even if I click click on every playlist and checked MAME as Parent, the playlist isn't added under "MAME".  And the playlist disappeared! How can I solve this issue? 

I also noticed that during the import process, it duplicates all the playlist.  Now I'm able to add under MAME oly "Pinball Arcade":

I tried to remove duplicate playlist and remove the word "Arcade" before every playlist name. But it doesn't help.  I noticed that if i remove some duplicate playlist, LB remove both the playlist", So fo example now I don't have 2 Players playlist, and so on...

How can I fix this mess? 

thank you!

 

EDIT: I closed and opened again LB, and now it correctly moves evey playlist under MAME. Probably there was an issue and rebooting LB it as solved.  But for sure I deleted some default playlist /2 Players, etc...one). How can I re-create all the missing playlist? Have I to delete again ALL the games, import again the fullromeset and again modify the parent of every playlist, removing the duplicates? Currenlty I have these playlist but I don't know which are missig (2 Players for sure):

thank you

Edited May 26, 2020 by legolas119

[Jason Carr]

@legolas119 Tools > Create Missing Arcade/MAME Playlists

[legolas119]

ah ok! thank you, very useful!!

[mcfilmmakers]

As suggested, I will continue the thought brought up in the 11.17 thread here.

The catver.ini on progretto snaps is in fact currently updated to 226 while Launchbox appears to still be at 224. Because Launchbox does not communicate this discrepancy to users, it is a problem, especially when the intent is to update Launchbox's files a couple of days after each Mame update.  The point here is simply to highlight that users are being asked to wait un unknown amount of time to update to unknown versions of Mame at irregular intervals independent of Mame updates.

 

In short, the crux of the issue is users can't import ALL the roms they want. if the user is at 225 but Launchbox is at 224, the user will never know that the 225 games did not import (unless they paid attention). It's only worse if you're trying to import 226 as you'll never know about either 225 or 226 missing roms.

The only alternative that permits the import of ALL the roms they want, is to import without Mame as the emulator, but then the games' titles match their filenames, which is useless and a HUGE time sync to rectify manually. There has to be a compromise possible.

The suggestion is to 1) communictae the currently supported mame set in LBDB and 2) allow users to use their own dat files, at the very least to import ALL the roms they want using the dat file's titles. The rest of the metadata can wait for the launchbox DB update if necessary (though a lot of the info should be in the dat file anyway).

This can be done in the mame importer (before it asks if you want to skip certain types of games) with a screen which states what the current LBDB version is with a field asking if the user wishes to proceed using the LBDB version or use their own dat file. The software list plugin does a good job doing this for reference.

 

Edited November 9, 2020 by mcfilmmakers

[Jason Carr]

On 11/8/2020 at 12:48 PM, mcfilmmakers said:

As suggested, I will continue the thought brought up in the 11.17 thread here.

The catver.ini on progretto snaps is in fact currently updated to 226 while Launchbox appears to still be at 224. Because Launchbox does not communicate this discrepancy to users, it is a problem, especially when the intent is to update Launchbox's files a couple of days after each Mame update.  The point here is simply to highlight that users are being asked to wait un unknown amount of time to update to unknown versions of Mame at irregular intervals independent of Mame updates.

 

In short, the crux of the issue is users can't import ALL the roms they want. if the user is at 225 but Launchbox is at 224, the user will never know that the 225 games did not import (unless they paid attention). It's only worse if you're trying to import 226 as you'll never know about either 225 or 226 missing roms.

The only alternative that permits the import of ALL the roms they want, is to import without Mame as the emulator, but then the games' titles match their filenames, which is useless and a HUGE time sync to rectify manually. There has to be a compromise possible.

The suggestion is to 1) communictae the currently supported mame set in LBDB and 2) allow users to use their own dat files, at the very least to import ALL the roms they want using the dat file's titles. The rest of the metadata can wait for the launchbox DB update if necessary (though a lot of the info should be in the dat file anyway).

This can be done in the mame importer (before it asks if you want to skip certain types of games) with a screen which states what the current LBDB version is with a field asking if the user wishes to proceed using the LBDB version or use their own dat file. The software list plugin does a good job doing this for reference.

 

I am aware of this issue. I think the only solution that would please everyone at this point is to provide a power user feature where users would have to manually go and download files from around five different locations, and then LaunchBox would combine them together and build a custom database. I really wanted to avoid that, and I still do, but that's the only way to allow users full control over that process. We could certainly figure out a way to notify users of what MAME version LaunchBox is currently using, but I sincerely doubt that will be enough for people. Most likely, we'll put this on the next poll.

[mcfilmmakers]

15 minutes ago, Jason Carr said:

Most likely, we'll put this on the next poll.

Totally fair.

Notifying users of the supported version will go a long way of helping users to avoid potentially updating too far. It would certainly help on support issues by simply pointing out "you aren't importing a supported version of MAME, you were told that when you imported the roms".

I do think though that a reasonable compromise for power users would be to allow import of the rom with the Game's title based on the Mame.exe hash (which JoeViking noted can be extracted from the exe). This way, users won't have to pull a bunch of files from everywhere, just the mame.exe I'm fairly certain more than just the game's title can be pulled from that too. With the user's understanding that the import is basically just a straight import of the roms without sorting them or filtering. User's basically would use the power import to let them import 100% of what they selected, with the correct title (and potentially other possible metadata) for MANUAL sorting after import. At first, anyway.

Anything else that can't can be relegated to "please wait for the Launchbox DB to update then run the metadata wizard" if necessary but at least users will be able to set up (and play) in the meantime and get collecting on their images/videos. Perhaps an ability to "flag" your platform during the import process to have Launchbox notify you when the Mame version is updated would be an asset here (to avoid questions of "when will it be updated?"). The check can be made when launchbox launches at the same time that it checks if a new version of launchbox is available. Just a thought.

Edited November 11, 2020 by mcfilmmakers

[Jason Carr]

6 minutes ago, mcfilmmakers said:

Totally fair.

Notifying users of the supported version will go a long way of helping users to avoid potentially updating too far. It would certainly help on support issues by simply pointing out "you aren't importing a supported version of MAME, you were told that when you imported the roms".

I do think though that a reasonable compromise for power users would be to allow import of the rom with the Game's title based on the Mame.exe hash (which JoeViking noted can be extracted from the exe). This way, users won't have to pull a bunch of files from everywhere, just the mame.exe I'm fairly certain more than just the game's title can be pulled from that too.

Anything else that can't can be relegated to "please wait for the Launchbox DB to update then run the metadata wizard" if necessary but at least users will be able to set up (and play) in the meantime and get collecting on their images/videos. Perhaps an ability to "flag" your platform during the import process to have Launchbox notify you when the Mame version is updated would be an asset here (to avoid questions of "when will it be updated?"). The check can be made when launchbox launches at the same time that it checks if a new version of launchbox is available. Just a thought.

It's the filtering and combining processes that require the extra metadata, and those processes are arguably the most important features of the MAME importer, as no one wants all those duplicates just thrown around in their collection. So I don't consider it a possibility at all to provide any of this functionality with just the MAME dat file. It's just wouldn't be very useful, and it'd be cause for massive confusion (as if there isn't enough confusion with it already).

[mcfilmmakers]

Just now, Jason Carr said:

It's the filtering and combining processes that require the extra metadata, and those processes are arguably the most important features of the MAME importer, as no one wants all those duplicates just thrown around in their collection. So I don't consider it a possibility at all to provide any of this functionality with just the MAME dat file. It's just wouldn't be very useful, and it'd be cause for massive confusion (as if there isn't enough confusion with it already).

I hear you. You know the process better than anyone!

In that case, providing the supported version on import is a great 1st step. Allowing a power user import option is an excellent step from there, all it would take is a mention of filenames required and asking the user to make sure the files are all in the same folder, ask the user to point to the folder and done. I think it sounds like a very simple process (from a UI standpoint). Yes, collecting the files might be inconvenient but the user wanting and using this feature wouldn't be complaining about it anyway and frankly, isn't foreign to anyone who bothered collecting roms in the first place. The files required for the import don't carry the same legal or availability hurdles as the very roms being imported  anyway. Heck, Launchbox may even be able to point users directly to progretto (or elsewhere) to get them.

[legolas119]

yes I'm agree, it could be very useful to know which is the current MAME version supported in LB: in this way we can know when we can update the romset in LB

Do you confirm that the current supported version is 0.224?

 

thanks