Content download/Import speeds - 16hrs so far.

[Rygar]

Before I get bombarded with RTFM etc, I did investigate if others were having this issue for a few hours and there is little or no response on reddit posts or this forum or the LaunchBox website.

Most responses seem to be "That's not right", "I've got nothing."

Example:

Now I'm at 16 hrs so far for this much, considering my i7 9700k, EVO m.2 drive, hard-drive space and 100mb/s internet are not the issue, it has to be on the server side. I'm in Australia (Brisbane).

Can I cancel this process and not have to go through the WHOLE thing again???

I've tried turning off windows security, I've excluded the LaunchBox and Mame folder, I am now at a loss on what in my system could be slowing this (but not blocking it). The only conclusion is the import process itself is extremely slow. What exactly happens in this process, which servers are used to download content etc. Emuline doesn't appear to be slow or down.

This has to be one of the slowest content delivery methods I've ever encountered.

Any suggestions?

EDIT: OK I cancelled the import, kept the files so far and they show up. It's just an abysmally slow process I guess, any suggestions for improving the speed for the rest still welcome.

Edited November 14, 2020 by Rygar

[neil9000]

2 hours ago, Rygar said:

Before I get bombarded with RTFM etc, I did investigate if others were having this issue for a few hours and there is little or no response on reddit posts or this forum or the LaunchBox website.

Most responses seem to be "That's not right", "I've got nothing."

Example:

Now I'm at 16 hrs so far for this much, considering my i7 9700k, EVO m.2 drive, hard-drive space and 100mb/s internet are not the issue, it has to be on the server side. I'm in Australia (Brisbane).

Can I cancel this process and not have to go through the WHOLE thing again???

I've tried turning off windows security, I've excluded the LaunchBox and Mame folder, I am now at a loss on what in my system could be slowing this (but not blocking it). The only conclusion is the import process itself is extremely slow. What exactly happens in this process, which servers are used to download content etc. Emuline doesn't appear to be slow or down.

This has to be one of the slowest content delivery methods I've ever encountered.

Any suggestions?

EDIT: OK I cancelled the import, kept the files so far and they show up. It's just an abysmally slow process I guess, any suggestions for improving the speed for the rest still welcome.

Well first, apologies for this, this certainly isnt normal behaviour, but could be due to any number of reasons, most of them beyond our control and just the nature of the internet.

We had someone in our Discord server who was also from Australia and he was getting really bad speeds and thousands of errors when downloading metadata and images, he was on Mobile 4G internet it turned out and as soon as he used a VPN everything improved (essentially his isp was blocking him from making thousands and thousands of requests to there servers) are you on mobile internet or just normal cable to the home?

2 hours ago, Rygar said:

The only conclusion is the import process itself is extremely slow

Well that is not the case for 99.9% of users, but we do occasionally get reports of slow import downloads from users, and we have no idea as to the cause. I mean obviously we have no control over the internet in general and the route through the internet that your data takes, the only thing we can physically do is make sure our servers are up to snuff, and they are as they max out most peoples connection. The way the importer works is that it is making often dozens of different requests for media for every game, every single image is a separate web call and this WILL take longer than than a standard single file download just because of the nature of thousands, if not tens of thousands of different internet calls being made. Our servers are based in the Netherlands, so being in Australia you do have a long route to the servers, but it shouldn't be much slower than anywhere else. Although 16 hours is definitely excessive though, but depending on how you imported MAME and the images/videos/manuals etc you requested that may not be beyond the realm of possibility, if you imported everything and not just working games you could easily be making over a 100,000 different requests to the servers, and requests themselves take time as well as the physical download of the files requested.

Id look online on how to do a trace test to see the route your traffic is taking and if there are any hiccups or holdups anywhere in that process. Id also try with or without a VPN active to see if that makes any difference (if its quicker with a VPN its likely a issue with your ISP rather than our servers).

[Rygar]

Thanks for the extensive response. I don’t have a VPN so I think I’m at the mercy of the internet gods.

It was 2/3 done, I canceled and kept the imported content hoping to continue the download overnight tonight. Unfortunately it seems to start from the beginning again rather than continuing, so I’m hoping I can figure out how to skip already downloaded content. I used the default option for importing the full Mame pack in the beginning and that included a lot of options.

Does skip clones on import mean skip already download content or does it mean skip cloned/bootleg versions of games?

Edited November 14, 2020 by Rygar

[neil9000]

6 minutes ago, Rygar said:

Thanks for the extensive response. I don’t have a VPN so I think I’m at the mercy of the internet gods.

It was 2/3 done, I canceled and kept the imported content hoping to continue the download overnight tonight. Unfortunately it seems to start from the beginning again rather than continuing, so I’m hoping I can figure out how to skip already downloaded content. I used the default option for importing the full Mame pack in the beginning and that included a lot of options.

Does skip clones on import mean skip already download content or does it mean skip cloned/bootleg versions of games?

If you just ran the importer again it will by default grab all images again if they are checked. You should really just run the "update metadata and media" wizard from tools by highlighting all games in the platform with CRTRL-A, this will give you a option to only download missing media, but you would need the actual games imported first. What i would do is cancel what you have now and run the importer and UNCHECK all the media downloads (leave the download metadata checked). This way the actual import of the files and the metadata will only take a few minutes, you can then worry about artwork after by using the previously mentioned tool that has more options available.

[Rygar]

Thankyou, this sounds like a plan. 

I tried a traceroute, I'm getting an insane amount of timeouts which may explain. I'll do what you say and leave it to continue overnight. Thanks again for your help.

EDIT: Ok wow, the roms and meta data imported in literally less than a minute. So the slowdown is definitely all the content. Thanks again for your help. Hopefully this can assist others that have a similar issue.

Edited November 14, 2020 by Rygar