Jump to content
LaunchBox Community Forums

Importing is so slow


Recommended Posts

It's taking over 8 hours so far to import 9000 SNES games. I have to do this for every platform. Can they work on massively speeding up the import process? It's taking so long as has been like this for ages I'm now looking at alternatives. 

Link to post
Share on other sites

What is your existing setup size, how many systems and games do you have already? If you have a lot as I am guessing this will be the cause. If you import on an empty LaunchBox setup it will be almost instant.

Conclusion in this thread

Edited by wallmachine
Link to post
Share on other sites

If you are importing alot of games at once and downloading media make sure you uncheck to box that references “rough matches” for media. Having this checked will slow down large imports as it is comparing every image. 

Link to post
Share on other sites
6 hours ago, NOMAD0P said:

9000 SNES games

A No- Intro SNES set is only around 3500 files including special chip dumps. I don't know what rom set you are trying to import but if i had to guess I would say it is a GoodSet which has many duplicate games.

Link to post
Share on other sites
11 hours ago, NOMAD0P said:

It's taking over 8 hours so far to import 9000 SNES games. I have to do this for every platform. Can they work on massively speeding up the import process? It's taking so long as has been like this for ages I'm now looking at alternatives. 

Importing 9.000 games all at once may take a few hours for sure. 8 hours seem a bit long to me, but I've never imported this many games for a system with as much metadata and media as the SNES, so it may just be that it really takes this long.

I have a no intro 1G1R SNES set. I believe it has just short of 2.000 roms. Importing that and downloading all the media on the LB database as well as emumovies took a very long time for me as well. I'm not sure exactly how long, but I think around 2 hours? I'm really not sure anymore. It's been a while since I have updated the set.

Link to post
Share on other sites

Yeah im not even sure if that could be classed as particularly slow. You have 9000 roms (completely overkill as likely a "good set" with tons of duplicates and broken games, a complete waste of time and space, but whatever), and SNES is a popular platform with a ton of art for it. Lets do some quick maths, if every game had 10 images (most of them will probably have far more than that) then you are downloading 90,000 individual pieces of artwork, which all need there own separate and individual call to the servers, its not like we can just offer you a zip of only the images you specifically need or want, as there would still need to be all those server calls in order to get a list of what is available to you, and then package that into a single zip file, but then what happens if your internet goes out or the zip gets corrupted somehow on download, you then have a massive zip file that goes straight in the bin. The way it is currently done is the best way.

Bear in mind also the size of some of the images you are downloading, fanart for example can be rather large in size, and some games can have 20-30 different fanart images to download, that quickly adds up. Also i assume you are scraping emumovies as well? Cause then you are sending all of these individual server requests to two servers, ours and theirs, and there servers are slower than ours, so that will slow things down also.

Looking at my SNES artwork folder for my around 850 SNES games i have, my folder is nearly 5GB, and i dont download all art types, or music, or manuals (which can be hundreds of megabytes each themselves). You have ten times the amount of games as me so again doing the maths you would be downloading around 50GB of images, and that is without manuals, music, and videos, as well as specific image types i dont have myself. I dont know how fast your internet is, but on mine if that was just a single zip file that would take me nearly 3 hours to download on my connection, 90,000 and probably much more than that different files are gonna take much much longer than that.

Link to post
Share on other sites

Same problem here 😞 I was importing a full NES set on a fresh install (latest version of LB). I started it at 8pm and the next day 10am only 40% was done. Importing a single game from this set is taking more as 5 min each. NOt sure if this a problem with th elatest version but on my other PC this isnt a problem (older version).

Link to post
Share on other sites

Okay so glad it's not just me. I have a brand new pc with a fresh windows install, 16gb ram and a high powered processor connected to a gigabit ethernet. My 790+ NES roms took about 8 hours and my SNES which is like 2400 or so are at 14 hours now. I'm not even doing Emumovies. As a new user who just found LB this is pretty disappointing. 

Link to post
Share on other sites

Rough Image match can cause the media import process to take quite a bit of time. It is best to keep it turn off until you have a good baseline established and are only using the media download tools to fetch a few new images from the database.

Link to post
Share on other sites
On 2/28/2021 at 7:17 AM, neil9000 said:

you are sending all of these individual server requests

This needs to be repeated.

Downloading so many different smaller files takes much longer than people think it should. Each file is a new request to the server which tends to take more time than the actual download time of the file. You see this on your computer as well if you have ever copied a large folder of small files versus copying a single large file, the folder with a lot of small files takes much longer to copy.

Also bear in mind we have seen issues with users on certain ISPs, especially from parts of the world like Australia or certain American ISPs.

If you want a quick import just scrape the basic files you need first like box art and gameplay screenshots. Leave the stuff like videos and fan art for a later time. You can always go back in and re-scrape for that media only at a later time.

Link to post
Share on other sites
4 hours ago, C-Beats said:

Rough Image match can cause the media import process to take quite a bit of time. It is best to keep it turn off until you have a good baseline established and are only using the media download tools to fetch a few new images from the database.

I stopped the import and turned that setting off. It made a small difference but not much. Thanks for giving me a suggestion to try.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...