Slow Import - There has to be a better (faster) way to do this?

[Gr33k]

Love Launchbox! Bought it and got lifetime/platinum (whatever) membership to emumovies and what not. That said - I am trying to set it up on a new retro machine I got last night. I am importing JUST the Mame and it's taking almost 8 hours and it's not even half done. This is just ridiculous in 2021 where we have massive amounts of bandwidth. For a paid product this is also unacceptably slow (I'm sad to say). How can the developers improve this (and soon, because I've seen posts about this in other places)? I can't imagine there isn't a way to improve this to under and hour (for just Mame for example) vs taking the better part of a day to complete...Either you folks need to upgrade your servers, and/or the software needs to be improved?

What can we do to improve this extremely slow (and frustrating) process folks? 

Thank you!

[neil9000]

1 minute ago, Gr33k said:

Love Launchbox! Bought it and got lifetime/platinum (whatever) membership to emumovies and what not. That said - I am trying to set it up on a new retro machine I got last night. I am importing JUST the Mame and it's taking almost 8 hours and it's not even half done. This is just ridiculous in 2021 where we have massive amounts of bandwidth. For a paid product this is also unacceptably slow (I'm sad to say). How can the developers improve this (and soon, because I've seen posts about this in other places)? I can't imagine there isn't a way to improve this to under and hour (for just Mame for example) vs taking the better part of a day to complete...Either you folks need to upgrade your servers, and/or the software needs to be improved?

What can we do to improve this extremely slow (and frustrating) process folks? 

Thank you!

Bandwidth should max your connection. If you are getting very slow imports then im going to guess you left the "remove roughly matched game images" option checked. That will hugely slow your imports down as every single image downloaded is compared via AI and the lower quality one is deleted. So i would look at your settings before you start bad mouthing a product and server infrastructure that you know nothing about.

[Gr33k]

Hi Neil! Thank you for the super fast response.

Why is this option 'recommended' if it's hindering the performance so much (just wondering hehe)? Should be disabled by default perhaps. I cancelled the import and started it over again and indeed it is MUCH faster.

Is it too much to wonder "Can it still get any faster?". I'm on 500/500Mbps on a decent spec'd PC. What can we do to make this faster? Multiple threads possibly running in tandem?

Ty

UPDATE - The initial part was faster, but as soon as it caught up to the new items not imported yet, it was still kind of slow to complete (still better but I can't help but want to brainstorm to see if we can do something better!)

Edited June 7, 2021 by Gr33k

[Lordmonkus]

This subject gets brought up fairly often and the answer is always the same.

It's not your internet speed or the servers connection, it's that every single image or piece of data downloaded is a single request that your computer needs to send and the server needs to respond to. This is where the bottleneck is.

On your PC try copying a single large file versus copying many thousands of smaller files. You will find the single large file takes far less time. This is the same thing that is happening here.

The alternative is to download a complete media pack from some site and hope they are all named correctly for your collection and then go through it all manually to fix up any errors.

[neil9000]

And to add to the above remarks the downloads already are multi threaded.

[Gr33k]

If downloading files one-by-one is the bottleneck - let's get around that

The multiple files do take much longer. Packs - that sounds interesting. Would it be feasible to provide a collection pack for download? Most of the consoles are not producing new games of course - would it be better to provide an option for a full pack download (automatically via LaunchBox), and let the local machine post-process everything vs doing it one by one over a network connection? Like MAME - a pack of all the (playable) games...perhaps even easier NES (USA/JP/EU could be separate packs or not...), SNES,Sega, etc. I'd prefer to download the basics from the pack, and emumovies if I want the videos separate. That would still be faster than one by one lookup of each media type.

If wasted space is a concern - Perhaps prune un-used media that isn't associated with games a user has (at their discretion)?

What do you guys think?

Edited June 7, 2021 by Gr33k

[Lordmonkus]

12 minutes ago, Gr33k said:

If downloading files one-by-one is the bottleneck

No, the server requests are the bottleneck.

12 minutes ago, Gr33k said:

What do you guys think?

I think let Jason do it how he feels is best.

I've tried the "media pack" method in the past with other front ends and it sucks big time. You spend far more time going through it and fixing it up than actual playing. I'd rather let the importer do its thing while I do other stuff than sitting in front of a computer for days on end going through it all manually.

[Gr33k]

21 minutes ago, Lordmonkus said:

No, the server requests are the bottleneck.

I think let Jason do it how he feels is best.

I've tried the "media pack" method in the past with other front ends and it sucks big time. You spend far more time going through it and fixing it up than actual playing. I'd rather let the importer do its thing while I do other stuff than sitting in front of a computer for days on end going through it all manually.

The pack would definitely need to be post processed on the local machine (which other solutions don't seem to do - they simply rely on the naming convention of files to match the pack's files..). Grab all the files, match them up to content that is present, then (optionally) prune any un-associated media files to save space (optionally - I personally might keep them). This could be a local database that is checked like a cache. Updating this content later online would be neat to still have, but for initial bulk import and creations - Something much faster is (obviously) desirable  

Just thinking of ways to get around a clear problem vs accepting it (that's just  how I am lol). Thank you for all your work on this great project!

Edited June 7, 2021 by Gr33k

[neil9000]

21 minutes ago, Gr33k said:

Just thinking of ways to get around a clear problem vs accepting it

It's not a problem though, it's just that your expectations are way out of whack with reality. We have a online games database, that by its very nature means it's online based. The other issue with packs is not everyone has full romsets for platforms, so they don't want or need the "full" artwork pack, so doing it that way would force people to download stuff they dont want for absolutely no reason. Also many many people all over the world have bandwidth and data caps imposed by there ISP's, you don't it seems, which is great for you, but not everyone else is in that fortunate position, so forcing them to download a ton of assets they do not need is really not a good direction to be heading in.

[neil9000]

Also two other things, not everyone wants all the different media types either, and a pack uploaded to the forums would be outdated the following day. So those are two other reasons why that's not a good idea.

 

[Gr33k]

46 minutes ago, neil9000 said:

It's not a problem though, it's just that your expectations are way out of whack with reality. We have a online games database, that by its very nature means it's online based. The other issue with packs is not everyone has full romsets for platforms, so they don't want or need the "full" artwork pack, so doing it that way would force people to download stuff they dont want for absolutely no reason. Also many many people all over the world have bandwidth and data caps imposed by there ISP's, you don't it seems, which is great for you, but not everyone else is in that fortunate position, so forcing them to download a ton of assets they do not need is really not a good direction to be heading in.

I would make it optional of course to download a full pack. Nobody has to be forced to do anything  There are a ton of options already available, I didn't think one more checkbox that offers "Download Pack (warning maybe large file size - or explicitly states the size of the file)" would hurt too much.

I don't think it's uncommon for someone to have a "Full NES USA Set" or "Full MAME 0.181" collection for example. I would think many users that do not have bandwidth caps would opt to download a pack and post-process it to match up with their content and save hours worth of time as a trade off. Am I the only one that leaves all of the boxes checked and lets the content roll in lol? I also have the emumovies sub so I can get the best video clips possible..which would be separate from the pack of course and also optional - I'd do this on the second run perhaps, or it will still be faster to download just the emumovies vs the rest of the content since one request per game. 

If they have the full NES/whatever set - the pack would cover it all. I don't see how a pack will be outdated when there are no future NES games being released (as one console but many of these would fall under the same umbrella for discontinued consoles). I also don't see why a person that wants to just download cover art would want the pack...it's to cover someone that has "all the boxes checked" that you folks make available as options in the application.

The reasons you point out DO make sense, but if I had a curated set of games - then odds are the download/import will be much faster than someone like me, for example, that has collections of each console. Those people will likely not be posting about how long it's taking to import their 50 NES, 100 MAME, and 25 Sega Genesis games, etc...or their Steam/GOG collection which is no where near my 100k+ game collection. If they also wanted less media types, that would speed up their look-up as well. I'm referring to the "I want the loaded package folks". And why shouldn't people? The options are there - If I'm making a nice arcade setup - I want all the flash and bells the software can provide  

I also have an accurate sense of expectations in that this is not going to be 'instant' by any means (I'm part of a couple of software development teams where I do this sort of thing, but I'm not a full blown programmer either - just a script kiddie and coding n00b that understands software development from a medium level perspective I suppose). However, I'm still not even 50% done with the MAME collection even with "remove roughly matched game images" Disabled. If server calls is a bottleneck, and multi-threading is already taking place - what else can we do to improve this process? Yes it's online, but if you download a pack and cache it locally, that takes away that bottleneck (at the expense of large downloads and local storage space) does it not? If we could make a few days down to a few hours that would be a significant performance gain.

Given that it takes literally days for me to import my collection AND to check for updates etc. I would be more than happy to take large packs and process everything locally vs always doing one by one look-ups online. Of course - I'm just looking for ideas on how we can overcome this challenge. I hope others might have some ideas as well perhaps. I do appreciate you brainstorming here with me  

Edited June 7, 2021 by Gr33k
typos ugh

[dragon57]

14 hours ago, Lordmonkus said:

The alternative is to download a complete media pack from some site and hope they are all named correctly for your collection and then go through it all manually to fix up any errors.

This is what I do now. I get the combined box, box 3d, snaps, title, wheel packs from EmuMovies, then manually check for missing art. I have numerous scripts to do compares and setup for art downloads. Using the EmuSync tool from the EmuMovies site also seems to be faster than LaunchBox in some cases in downloading art, but everyone's use case will/may be different. You just have to find what works best for you with the tools and assets available to you. You also need to revisit all alternatives when new versions come out.

The LaunchBox devs are doing their best for us I feel.

As to your thoughts on downloading a 'full pack' from the LaunchBox database/art server, I feel this could be done in code, but it would greatly increase the horsepower demands on the online server. The process would be similar to what happens if you have a Google Drive folder full of files that you want to download as a complete zip archive. The Google server creates a zip on the fly of the files you selected. In my experience, the Google process can take a long time to prepare an archive of many hundreds/thousands of files.

 

[zugswang]

12 hours ago, Gr33k said:

I would make it optional of course to download a full pack. Nobody has to be forced to do anything  There are a ton of options already available, I didn't think one more checkbox that offers "Download Pack (warning maybe large file size - or explicitly states the size of the file)" would hurt too much.

I don't think it's uncommon for someone to have a "Full NES USA Set" or "Full MAME 0.181" collection for example. I would think many users that do not have bandwidth caps would opt to download a pack and post-process it to match up with their content and save hours worth of time as a trade off. Am I the only one that leaves all of the boxes checked and lets the content roll in lol? I also have the emumovies sub so I can get the best video clips possible..which would be separate from the pack of course and also optional - I'd do this on the second run perhaps, or it will still be faster to download just the emumovies vs the rest of the content since one request per game. 

If they have the full NES/whatever set - the pack would cover it all. I don't see how a pack will be outdated when there are no future NES games being released (as one console but many of these would fall under the same umbrella for discontinued consoles). I also don't see why a person that wants to just download cover art would want the pack...it's to cover someone that has "all the boxes checked" that you folks make available as options in the application.

The reasons you point out DO make sense, but if I had a curated set of games - then odds are the download/import will be much faster than someone like me, for example, that has collections of each console. Those people will likely not be posting about how long it's taking to import their 50 NES, 100 MAME, and 25 Sega Genesis games, etc...or their Steam/GOG collection which is no where near my 100k+ game collection. If they also wanted less media types, that would speed up their look-up as well. I'm referring to the "I want the loaded package folks". And why shouldn't people? The options are there - If I'm making a nice arcade setup - I want all the flash and bells the software can provide  

I also have an accurate sense of expectations in that this is not going to be 'instant' by any means (I'm part of a couple of software development teams where I do this sort of thing, but I'm not a full blown programmer either - just a script kiddie and coding n00b that understands software development from a medium level perspective I suppose). However, I'm still not even 50% done with the MAME collection even with "remove roughly matched game images" Disabled. If server calls is a bottleneck, and multi-threading is already taking place - what else can we do to improve this process? Yes it's online, but if you download a pack and cache it locally, that takes away that bottleneck (at the expense of large downloads and local storage space) does it not? If we could make a few days down to a few hours that would be a significant performance gain.

Given that it takes literally days for me to import my collection AND to check for updates etc. I would be more than happy to take large packs and process everything locally vs always doing one by one look-ups online. Of course - I'm just looking for ideas on how we can overcome this challenge. I hope others might have some ideas as well perhaps. I do appreciate you brainstorming here with me  

Try using EmuMovies Sync for the videos , logos etc if you have EmuMovies as it's very fast, even with small single items like logos, snaps, videos etc

You can then untick these from your main search via Launchbox which will speed up your overall time.

[neil9000]

3 hours ago, zugswang said:

Try using EmuMovies Sync for the videos , logos etc if you have EmuMovies as it's very fast, even with small single items like logos, snaps, videos etc

But do bare in mind that naming may be different there so may not line up with any of your games. When downloaded from emumovies from within Launchbox we rename files on the fly when needed. Going the manual route obviously wont do that.

[zugswang]

14 minutes ago, neil9000 said:

But do bare in mind that naming may be different there so may not line up with any of your games. When downloaded from emumovies from within Launchbox we rename files on the fly when needed. Going the manual route obviously wont do that.

Yes, but they all still get picked up correctly by Launchbox, as I do this all the time.

[Gr33k]

To be clear - I didn't even setup EmuMovies yet on this PC. I just imported my MAME collection with all the defaults enabled (then removed the roughly matched images which didn't speed it up too much, but a little). It took many hours so I just left it going overnight and found it completed this morning. 2108 games or so...took multiple hours to import some images... Maybe that seems acceptable to some folks here and that's "the best we can do"...but I think it could be much better if we tried to find a solution vs accept the slow garbage as being the only possibility. Maybe the server needs to be better/faster/capable.

Perhaps HAProxy/load balancer to multiple servers - I'm not exactly sure because I don't know how it's currently setup. Maybe it's just me perhaps and my expectations are "out of whack" (many people are complaining about the same thing, so maybe it's the dev's expectation that is out of whack lol). What I can say is it sucks how long this takes and I wish it was much faster lol. Any time I want to 'check for updates' on media (which I wouldn't if I had a pack that had it all in there perhaps), it also takes multiple days on my main machine.

EDIT: And by packs I mean LaunchBox hosts them and maintains them - not a 3rd party.

 

Edited June 8, 2021 by Gr33k

[neil9000]

2 minutes ago, Gr33k said:

Maybe that seems acceptable to some folks here and that "the best we can do"...but I think it could be much better if we tried to find a solution vs accept the slow garbage as being the only possibility. Maybe the server needs to be better/faster/capable.

If you keep up with this derogatory tone you will end up with a temporary or permanent ban. We have thousands upon thousands of users here that have no issues with our servers, if they are not up to your standards then your standards are completely unreasonable.

[Gr33k]

Thanks Neil - I see where you are coming from now. You are happy with the way it is regardless of how many people complain about it. That's all you have to say vs tell me my expectations are out of whack or threaten me with a ban.

I wonder if they are paid members that support the project like I am? Regardless - Glad they are happy with your software's performance. I'm not. Last post from me as this has been pointless and super unhelpful.

Edited June 8, 2021 by Gr33k