Lightspeed: Ultra-Fast Mame Importer Tool

[Drybonz]

2 hours ago, Antropus said:

Not so fast! I am very busy at the moment, but I didn't completely quit yet

-Kris

That's good to hear.  I was hoping this project would stay alive.  Thanks.

[SpaceMidget75]

Long time C# dev here. Both solutions are great so you should all be pleased with what you've achieved thus far. Thank you.

@Antropus for what it's worth LB running with a Mame Set imported via your tool seems perfectly fine speed wise, running an i5 6600, SSD & 16GB. Maybe drive speed is the issue here? I'm also guessing that the XML changes should be minor, but after 15 years in the business I'm not going to assume anything without seeing the code, and as this is a personal project I completely understand that you need to determine when (and if) you're willing to make those changes. Someone else could always pick-up the project for you if real life does get in the way. It could even be integrated into the Launch Box Mame importer

@Jason Carr Keep up the good work! I love the frequency of new features and personally haven't seen it cause a negative impact on performance or stability yet.

[timekills]

I like lamp.
[emoji12]

Sorry, just enjoying the dev talk. I'd love to assist but I'm pulling hair out trying to finalize some projects for my own job.

[Jason Carr]

Thanks @SpaceMidget75; that's great to hear. Also it's always great to have other developers here in the community. @timekills It's been a while but I've used LAMP quite a bit as well.

[Antropus]

If the new XMLs were the only problem I would be done by now!
The program as you know it is no longer, so I'm not updating that last version you guys are using. It's a brand new program at this point.
There were so many major changes that happened while I was transitioning into something more flexible, powerful and faster, that those new changes just add up to the insane amount of work necessary for me to get this rolling.
The entire Launchbox module was broken after that first transition, for example. Yesterday I managed to at least get it to work, although not perfectly yet.

There are many things pending, including refining the logic for a great number of new possible filters. What's already working as well, is a scan done over good part of the artwork folders at startup, which takes just a few seconds, so when your list is populated you have the exact idea of what artwork you currently have and what' missing. With that, I can now write something to export missing/haves, by artwork type, as lists, making it easier for anyone to complete their artwork collections.
There's also a new search tool which accepts unlimited number of parameters, so you can easily find games by a certain or multiple manufacturers, for example.

The next version will include many new features and the intention is to make it an all-in-one list generator to basically replace RomLister all together, exporting lists to all major FEs. But the most ambitious part is to make it into a cross-FE list mixer, meaning that once it's done, you should be able to import lists from all major front-ends, re-filter if necessary, blend them together and spit a new list to any FE of your choice. That way, people with libraries already set in other FEs should be able to load their lists, mix them all and generate a new Launchbox list for example.

But that will only happen when I have the time to sit my butt and do it, which is rare these days

About the speed, I wasn't referring to importing the games. That should be fast either way.
It's more about the performance I'm getting in my machine once in BigBox, with thousands of games in my library. The artwork loads a little sluggish for me compared to what I am used to seeing in my old setup.

About the code itself, I'm not actually a coder per se. I'm just a curious hobbyist
I got into developing my own tools once I started using Autohotkey to setup my arcade machine. I then started learning more and more about it, especially when it comes to handling of files and compared arrays, to the point that the entire program is written in Autohotkey script language, so not sure how that code would be useful in the context of Launchbox, but once I have everything in place (one day), documented and clean, understandable code, I will have no problem opening it up to any programmer with Autohotkey knowledge so we can eventually make it better.

-Kris

 

[SpaceMidget75]

@Antropus Just one quick point, in case you wasn't aware. Cleaning up the images in LB actually deletes the original images from the MAME extras folders. Bleeding obvious that it would when you think about it, but also a problem given that people tend to get their MAME artwork from outside of LB which means torrenting them down again. Wonder if it might be better to copy the artwork required for the custom list to a folder specifically for use in LB? Takes up more space, but at least it keeps peoples MAME data safe.

Just a thought.

[Drybonz]

It sounds good, Antropus.  Hope you are able to get it to surface.  Good luck with it.

[Antropus]

15 hours ago, SpaceMidget75 said:

@Antropus Just one quick point, in case you wasn't aware. Cleaning up the images in LB actually deletes the original images from the MAME extras folders. Bleeding obvious that it would when you think about it, but also a problem given that people tend to get their MAME artwork from outside of LB which means torrenting them down again. Wonder if it might be better to copy the artwork required for the custom list to a folder specifically for use in LB? Takes up more space, but at least it keeps peoples MAME data safe.

Just a thought.

Oh man! That's dangerous!!! MAME has its own structure and as you said, people download very large torrents to get all those extras in place. In my opinion Launchbox should not remove files outside of its own structure.

Any cleanup process started within Launchbox, in my opinion, should only affect files within Launchbox's own folder, unless a very big "WARNING!" popup is used, explaining clearly what's about to happen, followed by at least one confirmation window, to make sure the user knows what's about to happen.

I used similar messages and warning for the move roms feature in Lightspeed.

@Jason Carr, can you help with this? I understand your philosophy about keeping everything in relative paths under LB for portability. It works great for many systems, but when it comes to MAME and MESS (Software Lists), they already have their own structures and all major torrents respect that. There are over 100.000 games potentially supported by the current MAME and mirroring the artwork+videos between people's collections and Launchbox might not be an option for some people, especially those using small SSD drives to store things for extra performance. 

Extra warnings or simply constraining any file removal to Launchbox's own folder should solve the problem.

 

Thanks!

[Jason Carr]

There are unfortunately other issues here with setting LaunchBox image folders to locations that shouldn't be modified. Any time LaunchBox downloads new images of course they're added to the folder, or any time a user removes an image it's deleted from the folder, etc. I can understand the confusion though; perhaps we should make it clear that you shouldn't set your LaunchBox image folders to locations that you don't want LaunchBox to manage.

We've run into similar issues with people using the same image folders between various frontends, and we did add a workaround so that files weren't unnecessarily renamed, but at some point LaunchBox needs its own image folders that it can actually write to. If LaunchBox can't change the images in those folders, then it can't change users' collections, so not allowing images to be deleted is really not an option here.

The only solution would be to copy files to the folders instead of expecting that users will never modify the images in their collections.

[SpaceMidget75]

Yes, this is completely understandable. LB needs its own image folders that it controls and maintains. The only real conclusion to this is that a) IF people have pointed to external collections, sufficient warning is put in place and b) for the most part images from external sources should be COPIED to the default image location for the given system in LB.

As I said, this takes up a bit more space but allows LB to manage what it considers it's own setup.

@Jason Carr When you add images via the "Add Image" dialog (not download) does it keep the browse path or copy the image to LB? 

[Jason Carr]

Thanks @SpaceMidget75, agreed on all points. When you add images in the Add/Edit dialog they are indeed copied. The originals are left in place.

[Johnnydement]

1 hour ago, SpaceMidget75 said:

Yes, this is completely understandable. LB needs its own image folders that it controls and maintains. The only real conclusion to this is that a) IF people have pointed to external collections, sufficient warning is put in place and b) for the most part images from external sources should be COPIED to the default image location for the given system in LB.

As I said, this takes up a bit more space but allows LB to manage what it considers it's own setup.

@Jason Carr When you add images via the "Add Image" dialog (not download) does it keep the browse path or copy the image to LB? 

Well, I see it as simple as having different "sources" folders and "work"...

For instance, I can point to several folders where I do have media, and LB downloads what's still missing on it's "work" folder...

Like mame itself, where you can point various folders for each concept...

In my case for eample, I do have LB and emus in an SSD, and roms and mame extras in a NAS... I could take great advantage in keeping these extras on the NAS and not mix with LB goods...

Edited September 27, 2016 by Johnnydement

[Antropus]

I understand that Launchbox downloads new images from the database straight to any folder you define, correct?
If I remember correctly, images I drag-dropped from a browser into the edit window/image were correctly copied into my Mame\Flyers folder, so I know it works just fine.
Lightspeed was designed to be a straight forward importer, using MAME's own folders do import large libraries into Launchbox, thus, consolidating images across the board, because it just makes no sense to have the same images saved into multiple paths, IMHO.

I do have multiple FEs, MameUI, Multiple versions of MAME and Launchbox, all pointing to the exact same image folders when it comes to arcades, but even for consoles, with the integration of MESS into MAME, I'm running several console systems and using artwork widely available in the form of Software Lists packages for MAME.

It might be convenient from your point of view of developing and managing, so I understand your reasons, but it's definitely not a flexible approach if you are expecting users to use LB's folders to save their artwork at all times

What I am gathering is that maybe there's another reason other than portability and I believe it has to do with LaunchBox's DB.
I remember reading posts in the past and concerns were raised about the possible consequences of adding images from other sources into the LaunchBox DB, like the possibility of having people downloading stuff from Emumovies and uploading them into Launchbox DB.

Then some posts about the intention to have people's libraries all connected to the DB, so they can upload/download artwork.

If that's the case, that you are trying to keep the DB clean from contamination of images coming from other sources, other than produced by users, then I understand your concern
As a regular Joe here though, I download images from any source that has them available (and there are many) and I believe other people do the same

-Kris

[Antropus]

25 minutes ago, Johnnydement said:

Well, I see it as simple as having different "sources" folders and "work"...

For instance, I can point to several folders where I do have media, and LB downloads what's still missing on it's "work" folder...

Like mame itself, where you can point various folders for each concept...

In my case for eample, I do have LB and emus in an SSD, and roms and mame extras in a NAS... I could take great advantage in keeping these extras on the NAS and not mix with LB goods...

Great point! Mame uses a simple ini where you can add multiple folders by separating them with ";". If Launchbox approached it similarly, then only images not found in any of the listed folders would be downloaded and this way Jason could keep absolute control over LB's image folders, sharing only those with the community via DB connection. Food for thought

Edited September 27, 2016 by Antropus

[Jason Carr]

Yes, it does sound like the best solution for the future here would be to allow you to add additional folders to parse without changing the default folder that is used for saving. That complicates a lot of things (I cringe at the thought of how the platform folders dialog is going to look), but I agree that that would be a good thing to add. Good thinking @Johnnydement.

[Antropus]

1 minute ago, Jason Carr said:

Yes, it does sound like the best solution for the future here would be to allow you to add additional folders to parse without changing the default folder that is used for saving. That complicates a lot of things (I cringe at the thought of how the platform folders dialog is going to look), but I agree that that would be a good thing to add. Good thinking @Johnnydement.

I was thinking the same! So many fields now with all the new artwork folders added. But if you kept everything exactly the same and simply added support to parse each field using ";" as a separator, for example, things would look virtually the same, only with longer lines for those who want to use multiple paths and exactly the same for those who don't

[Jason Carr]

Good point. Only problem with that though is we'll be back to some of the exact same issues because people won't know how to use it properly out of the box. Though it would fix the issue for Lightspeed.

[Antropus]

Not really an issue for Lightspeed as users can point it to any folder they want
It's just easier to use a structure established by MAME many moons ago and supported by people doing an incredible job for many years, like Mr.Do!, ProgettoSnaps, The "dome" etc

[Johnnydement]

Why not a "+" besides the "woirk" folder with a pop up to add extra ones? that would not crowd much default view...

[SpaceMidget75]

49 minutes ago, Jason Carr said:

Yes, it does sound like the best solution for the future here would be to allow you to add additional folders to parse without changing the default folder that is used for saving. That complicates a lot of things (I cringe at the thought of how the platform folders dialog is going to look), but I agree that that would be a good thing to add. Good thinking @Johnnydement.

Yep, a great UX is one of LB's strongest points and it wouldn't be wise to complicate the back-end configuration further with multiple folders IMHO. Maybe the following:

Launchbox should be aware if images are from it's own default folders or an external source.

Launchbox should warn you if you attempt to modify/clean images only if in an external source.

When adding images for a system it should ask if you want to copy the image to LB or link to it. (third party apps could also ask a similar question and either copy & set folder path or).

That way we're protected from accidental deletes, people can chose to maintain their images within LB or not (on a system by system basis) and I'm guessing it's not quite as a big change for LB and could be added to the roadmap easier than multiple/working folders.

Just brainstorming guys