MAME: how to update it in LaunchBox?

[neil9000]

8 hours ago, wallmachine said:

@sundogak as the LaunchBox team probably use the mame.exe to export a xml file for LaunchBox compatibility I suggested in the thread below they include it as a tool in LaunchBox so that the team do not need to constantly update maybe once every major release and let users decide in between, seems we've got multiple threads about MAME databases going on now.

 

 

The mame metadata file is not gotten from the MAME.exe. Its from a external source then converted to the correct format.

[legolas119]

hi,

well, I updated my LB to 10.15 yesterday evening, more or less at 10:00 pm (CET).  If I check my mame.xml file, it is updated at 01:01 am, so it seems that it was updated before to LB update:

I don't know why...

In any case, currently we are not able to understand if and when LB will update its mame.xml file in order to correctly run again the full MAME import process.

So, now, i could manually import the fullromset with the standard way (Tools --> Import --> Roms), uncheking all the Media. In this way, LB should automatically add the new roms and delete the not used. After that, I can use the standard function (Tools --> Import Metadata and Media --> Do not overwrite existing file) in order to download media only for the new added roms.

Am I right? I hope that in the future LB will clearly show which mame version is currently supported and will provide us an easy way to monthly update it

 

thank you

[DOS76]

No it doesn't matter how you import it either full set or the other way still uses the same list to get the working games list so if a game hasn't been added to that list by updating it to the new version then that game still won't get added.

[wallmachine]

59 minutes ago, DOS76 said:

No it doesn't matter how you import it either full set or the other way still uses the same list to get the working games list so if a game hasn't been added to that list by updating it to the new version then that game still won't get added.

With all types of importing options, MAME is the only one users can't control. Look forward to a possibility of it changing in the future or why the enhancement is not feasible

Edited May 20, 2020 by wallmachine

[neil9000]

30 minutes ago, wallmachine said:

With all types of importing options, MAME is the only one users can't control. Look forward to a possibility of it changing in the future or why the enhancement is not feasible

MAME by its very nature is different and requires a different import process. That is if you want Street Fighter 2 in your collection instead of sf2 with no metadata.

[legolas119]

1 hour ago, DOS76 said:

No it doesn't matter how you import it either full set or the other way still uses the same list to get the working games list so if a game hasn't been added to that list by updating it to the new version then that game still won't get added.

so the only way to correctly update MAME in LB is wait for a new MAME.xml version that consider the updated MAME.exe version. But unfortunately we can't know when this file is updated...

[neil9000]

7 minutes ago, legolas119 said:

so the only way to correctly update MAME in LB is wait for a new MAME.xml version that consider the updated MAME.exe version. But unfortunately we can't know when this file is updated...

Yes.

[Jason Carr]

On 5/8/2020 at 7:39 AM, Joe35car said:

So LB is like HS and we have to wait for a new database to be created on every new MAME release? Why can't LB create its own using the xml like MALA? 

Thanks 

LaunchBox has a much more involved algorithm for MAME imports. It's impossible to provide all of the options that we provide with just the MAME XML. The MAME XML does not have all the data needed in order to do the filtering and such.

The MAME XML that comes with LaunchBox is heavily processed and uses data from literally five different sources in order to provide the accurate data and filtering that the MAME full-set importer uses. It's not as simple as just using the MAME.xml file that comes with MAME.

18 hours ago, wallmachine said:

@sundogak as the LaunchBox team probably use the mame.exe to export a xml file for LaunchBox compatibility I suggested in the thread below they include it as a tool in LaunchBox so that the team do not need to constantly update maybe once every major release and let users decide in between, seems we've got multiple threads about MAME databases going on now.

 

 

Because of the above, the user would have to hunt down this data from five different places in order for any kind of a tool like this to work. We do have an internal tool that we use, but we make heavy use of SQL Server in order to process the data correctly. So that's not really feasible.

[Jason Carr]

9 hours ago, legolas119 said:

hi,

well, I updated my LB to 10.15 yesterday evening, more or less at 10:00 pm (CET).  If I check my mame.xml file, it is updated at 01:01 am, so it seems that it was updated before to LB update:

I don't know why...

In any case, currently we are not able to understand if and when LB will update its mame.xml file in order to correctly run again the full MAME import process.

So, now, i could manually import the fullromset with the standard way (Tools --> Import --> Roms), uncheking all the Media. In this way, LB should automatically add the new roms and delete the not used. After that, I can use the standard function (Tools --> Import Metadata and Media --> Do not overwrite existing file) in order to download media only for the new added roms.

Am I right? I hope that in the future LB will clearly show which mame version is currently supported and will provide us an easy way to monthly update it

 

thank you

We do need a good way to communicate with users what the current supported MAME version is, yes. The hardest thing that we have to deal with is the fact that we rely on several other data sources (such as catver.ini from Progetto-Snaps) and these are all manually updated by humans, after each MAME release. Some of them are updated sooner than others (some of them take a week or more to update, for instance).

What I generally do is wait a day or two after the official MAME release each month, and then update all the files that are available. It is possible to just update the data from MAME itself, and then the rest of the data can be updated when they're available. This means that for new games, the new games may be available, but might be missing some metadata until after the third-party sources update their files.

I know this seems like a big mess (and it is), but trust me, I put tremendous amounts of research into this when I built it; it's really the best option for accurate MAME data, and the code that we built to combine it all together is gold.

[Jason Carr]

Oh, also, no, you can't look at the date/time on the MAME.xml file in the Metadata folder to have any clue on when the MAME metadata was last updated.

[legolas119]

4 minutes ago, Jason Carr said:

We do need a good way to communicate with users what the current supported MAME version is, yes. The hardest thing that we have to deal with is the fact that we rely on several other data sources (such as catver.ini from Progetto-Snaps) and these are all manually updated by humans, after each MAME release. Some of them are updated sooner than others (some of them take a week or more to update, for instance).

What I generally do is wait a day or two after the official MAME release each month, and then update all the files that are available. It is possible to just update the data from MAME itself, and then the rest of the data can be updated when they're available. This means that for new games, the new games may be available, but might be missing some metadata until after the third-party sources update their files.

I know this seems like a big mess (and it is), but trust me, I put tremendous amounts of research into this when I built it; it's really the best option for accurate MAME data, and the code that we built to combine it all together is gold.

Hi Jason, but are you from the LB programmer team? well, your front end is absolutely fantastic  I have a lifetime subscription also for Big Box, that is really really cool. Thank you! 

Ok I understood for the MAME issue. So, for example, LB probably will be able to correctly import the additional roms for 0.221 full set in the next weeks? Usually LB could support the new MAME release after some weeks after the official MAME release date. Is it correct?

[neil9000]

29 minutes ago, legolas119 said:

Hi Jason, but are you from the LB programmer team?

Yes.

There are two coders currently on team, Jason, who is the creator and maintainer of Launchbox, and our new second dev, @C-Beats, who is doing awesome work (that i break constantly in betas and private )

 

 

[Jason Carr]

35 minutes ago, legolas119 said:

Hi Jason, but are you from the LB programmer team? well, your front end is absolutely fantastic  I have a lifetime subscription also for Big Box, that is really really cool. Thank you! 

Ok I understood for the MAME issue. So, for example, LB probably will be able to correctly import the additional roms for 0.221 full set in the next weeks? Usually LB could support the new MAME release after some weeks after the official MAME release date. Is it correct?

Yes, I'm the main developer and the original creator. Good to hear.

I just updated the MAME metadata for 0.221, but none of the third party sources are updated yet, so those are still on 0.220. You can go to Help > Force Update Games Database Metadata in LaunchBox to force a download of the updated MAME metadata.

 

[sundogak]

@Jason Carr Thanks for the info and clarification.   Understood on the versioning and experience that on the updates for graphics packages which lag versions by about 2 to 3 weeks from main MAME EXE.  Maybe just an XML header comment something similar to what you captured above?  Ex:  MAME Metadata = ver. 0.221  Third-party data sources = ver. 0.220   Anyway, appreciate all the updates/work on LB, its a great tool.

[TheNewClassics]

Hey Jason, thanks for explaining the process. With regards to the importer, any chance it can remember the location of where your MAME roms are?

[wallmachine]

4 hours ago, Jason Carr said:

We do need a good way to communicate with users what the current supported MAME version is, yes. The hardest thing that we have to deal with is the fact that we rely on several other data sources (such as catver.ini from Progetto-Snaps) and these are all manually updated by humans, after each MAME release. Some of them are updated sooner than others (some of them take a week or more to update, for instance).

What I generally do is wait a day or two after the official MAME release each month, and then update all the files that are available. It is possible to just update the data from MAME itself, and then the rest of the data can be updated when they're available. This means that for new games, the new games may be available, but might be missing some metadata until after the third-party sources update their files.

I know this seems like a big mess (and it is), but trust me, I put tremendous amounts of research into this when I built it; it's really the best option for accurate MAME data, and the code that we built to combine it all together is gold.

As we rely on the team to update the database for us it would be nice to be notified with some type of notification either via email, web page, in side LaunchBox, discord so we aren't left in limbo, yes I understand that some people do not care for the updates, but MAME has been around for decades and has a huge user-base and people always return to it (just over 1million people have viewed the CatVer page), why not try to capitalise on this. From the amount of threads/posts around here about MAME importing it/versions it seems a lot of people do not even know MAME is updated monthly. As we are talking about MAME here if people are actively notified that a new database has been imported they might be enticed to find out what does this update include? new games? fixes? new hi-score enabled games? oh awesome LaunchBox has a community leader-board for MAME too? what a great feature I might just buy the full version.

Yes, having the info in the xml file is useful but notifications send a bigger picture than users randomly guessing when to check this data has been updated and in general if you have to do it manually/guessing when to check you get tired of it.

For some time I have really wanted to launch the new Game & Watch/other new handheld titles via LaunchBox, and yes I can run them through MAME directly but then why did I purchase LaunchBox if I can't use LaunchBox/BigBox to play new released/fixed games.

 

Edited May 20, 2020 by wallmachine

[neil9000]

33 minutes ago, wallmachine said:

yes I understand that some people do not care for the updates, but MAME has been around for decades and has a huge user-base and people always return to it (just over 1million people have viewed the CatVer page), why not try to capitalise on this.

While i dont necessarily disagree with any of your points per se, you are really adding fuel to no real fire. Most people do not update there sets every month, in fact i would say a large proportion of our userbase are on much older sets and are perfectly happy where they are.

Im not saying i wouldnt personally myself like a bit more transparency as to when its updated, but you have to understand that people who update every month on the day its updated are a very very small proportion of the userbase. We have people using 20 year old sets still for example.

[MadK9]

I tend to agree with Neil here, i would like to see in LB just a versioning number for the current mame metadata version would be really nice, but i think there are far more important additions than this tbh.

Even though i do the no nags for mame each month, contrary to that my setup is still 0.205. Reading the patch notes each month tells you what new games/clones are added, and you can add them yourself quicker than the team here can update the data, its not hard to find the metadata, http://adb.arcadeitalia.net/ is one site that covers each rom with all the information you will ever need. 

[wallmachine]

On 5/8/2020 at 1:12 PM, sundogak said:

Here are my steps:

1. New MAME Set Arrives
   • Update your ROM location with your normal tools you use outside of LB, so complete version set (i.e., 220, 221). 
2. In LB
   • Select Tools >> Import >>MAME Arcade Full Set
   • Select Media - I personally turn off all media at this point.  See Step 3 for why I do it this way. If you want one stop done, select the media you want downloaded here. 
   • Select What ROMS To Import  - Option boxes, toggle which you do and do not want
   • LB will do its thing
   • Delete media as you noted if you have dupes/don't like.  I personally run LB media images with emuMoves video/manuals only settings. Then if no images in LB, run emuMovies via media download with just emuMovies to catch any missing.
   • Done.
     • MetaData Note: any data in the MAME XML file will always over-write any modifications you make if you use the MAME importer tool.  So things like Player Mode, Name, Date, Publisher, etc will always nuke whatever is there and replace it with the MAME data.  Where this is a bit annoying is for the Genre and PlayMode settings since MAME has completely different options there than LB. So using this tool, you somewhat have to live with the genre differences.  You can change them in LB, but if you reimport with fullset tool, you will lose all your changes since the XML takes priority.  Currently, there is no way to lock fields (wish there was!). 
     • MetaData Version Note:  LB can lag behind the monthly updates MAME (typically) does. I know of no easy way to tell which XML version each version of LB is using (seems like that should be something visible to me).  Since LB has been updated a lot lately this hasn't been issue. But if you download the new MAME set on Day 1, the XML is not likely updated and it won't import the game even if the romset has it physically in the folder.  However, you can force add a Game ROM via drag drop or Tools >>Import>>ROMs or alternatively wait until LB is updated with current set.  If want to see the XML used, it is in here:   ....Launchbox\Metadata\Mame.xml 
3. Optional: 
   • I hide some MAME games I have no interest in seeing, particularly ones with no media. If you just want a full complete set visible at all times, then most of this isn't applicable; hence optional.
   • I turn off media in Step 2 so I can see what "new" games are added and decide what want to do with them. By not downloading Media at Step 2 it makes easy to see. Monthly Arcade updates don't really add that many new games for most part so not like you have to wade through 100s of roms. 
   • For the ones I don't care about, I select them and move to a Playlist I call "ZZ-Do Not Use". Everything in this playlist I hide in LB.  The "ZZ" just keeps it sorted at end and out of way.
   • If you have games hidden, then in BB and LB they will not show up along with the associated playlist since BB doesn't show playlists where all games are hidden.
   • I don't worry about media at that point for hidden ones, since won't see them (most don't have media anyway, which is why started doing this to begin with).
   • For the few games I want visible, I download via the Tools >> Download Metadata and Media option (you can select all of them and do in one download batch then).  Alternatively, if you are getting MAME media updates, you can dump those in corresponding image folders since LB will pickup media with named with ROM.
   • Interestingly, with the MAME importer tool,  LB will "hide it again" when you do another monthly import with the Importer tool as long as you don't delete it.  So in my case, the ones I have hid reimport as "hidden" and are still in the Playlist(s) I setup.  That way I have fullset if want to see "all" the files but don't see them otherwise.
   • The reason for all this is that if you delete a MAME entry from LB, then when you re-import with the tool it will import them again (as it is designed to do). So you are back to having to hunt them down and removing each month.  If you hide, don't have to mess with them other than one time.  Also, if you delete an entry in LB, it loses its Playlist, Hidden, and LB specific metadata.

 

Hey mate,

• I had 4247 games under the MAME platform
• I followed your steps above and after the import it said 'imported 4273 games' however the actual amount under the platform is 4319 games
• I then deleted the entire platform and re-imported and it imported 4273 games and under the platform it shows 4273 games

Don't know where I went wrong and really don't want to delete it the platform each time as I think I started the re-import around 10am and now its 2PM almost... (No metadata scraping from LB DB and no image/video scraping)

Edited May 21, 2020 by wallmachine

[sundogak]

@wallmachine Yes, I didn't get into that in above description but suspect the numbers differed because of of way LB does imports if don't remove platform. So not something you did wrong.  The full proof method always is to delete the platform and re-import each time (I should have noted this).  If you don't do a platform delete then there is potential that you have games from prior selection in there (hence the 4273 where there were ~50 games "extra").  From what I can tell it doesn't keep anything where ROM is deleted from official set and no longer in XML.  But, if the game is still valid it will keep the game on a subsequent imports even if it was part of a set where you subsequently checked "skip" (hence why playlists stay intact).   

For example, I used this to my advantage in that I only wanted some of the Casino games, but didn't want to wade through them each time.  So I did a full import of casino games, deleted all the ones I didn't want (only from LB, roms leave them as is). Then on subsequent imports for monthly updates I check the "skip casino games" and it doesn't ADD any casino games I deleted back, but it also doesn't remove any casino games I left from the first import as long as they are still valid in MAME set.  That would be example of where numbers in my platform wouldn't match what the import numbers would be since those "others" remain.  So my guess is as you saw there are 4273 in the selection you made but before you did a full platform delete there were some "left overs" from within the "unplayable" or the "non-arcade" category....or....  

Or..... based on what Jason said in prior thread if the catver.ini and other INIs making up the LB MAME.XML lag behind each other (i.e., like now where main is at 221 and ini data is at 220) then you could get a glitch I hadn't thought about prior where the categorizations are not quite up to date and game is imported since doesn't know category yet.  But, it would only ever impact the (few) new games and if imported soon after MAME update and/or where the XML data isn't in sync with the INI categories.  A lot of edge cases for most people who don't update that often.

All conjecture, but I could see the numbers getting out of sync if doing like I noted above.  From my end, I typically go in right after import, hide anything that is new that don't care about and done. So if something came in accidentally then really I won't notice after that anyway.  But I could see it might confuse people with numbers not matching.  Ultimately, I am okay with any minor issues since the benefit is my import happens quickly and retain my overall setup with playlists, hidden, and favorites but something to be aware of. 

Edited May 21, 2020 by sundogak