Scraping is taking forever and constnatly stalling

[HomerJ]

I added some roms and I am trying to download all the media for them, but LB is constantly stalling out.  I have LB premium and EmuMovies premium.  Currenly it's been running for over 4 hours and it's on 56 of 24895. 

In fact, it looks like it's stuck.  It's been on 56 for a very long time now. 

Any thoughts?

Edited September 18, 2021 by HomerJ

[sundogak]

In my experience, trying to add that many at one time results in issues (particularly if videos are selected). My suggestion is to do in batches, for example by platform then do more if things work fine.  Additionally, I have found it better to match the metadata first (i.e., do search with no media download).  Use the Tools, Audit feature to verify things matched as you expect them to.  Then go back and have it search for other media. Videos in particular which are larger can cause hiccups and hold up rest.   Although sounds like more steps, I just have found more reliable.   Regardless, it shouldn't take 4 hours to do 56 games. So try what I suggested otherwise there is connection issue somewhere.  

Edited September 18, 2021 by sundogak

[HomerJ]

Thank you for the reply.

I will certainly try that in the future.

 

What I did after I posted this is turned off "remove similar images" and that seems to help.  If completed.

 

But now it came up with 6714 errors....since that is a different topic, I will start a new therad

Thanks again

Well, i was wrong.  The 6714 errors appeared during the first part where it updated the metadata. 

Now it is trying to downlod the files and stuck on file 2 of 24792.....stuck...

are there logs anywhere that I can see what is going on?

My Speed test is giving 900+mBps.  I am logged into EMU Movies and LB just fine.

I will try the batches way, but since this is getting stuck on file number 2 I have little faith.

Thanks again 

Edited September 18, 2021 by HomerJ

[HomerJ]

I just finished scraping about 29k items (not games, but all the meta data tha goes with about 4000 games).

However, when it completed there are 6714 errors (see screenshot).  so 23k or so scraped ok, but these 6714 have some access issue.

1) Should I keep the metadata updates despite these errors?

2) Any thoughts how to fix it?

Thank you!

 

Update, this was just the first step....metadata download.    Once i clicked yes, it is now stuck on files 2 of 24792...:( Back to the other thread.

 

Edited September 18, 2021 by HomerJ

[Retro808]

I merged your threads. There is no need to 2 threads with related issues.

Looking at your images path access is being denied. Any chance you have a folder set to "Read Only"?

[HomerJ]

OK, this is the Noobs forum.    How do I import metadata for just one platform?

 

[HomerJ]

Sorry.. figured they were different enough

14 minutes ago, Retro808 said:

I merged your threads. There is no need to 2 threads with related issues.

Looking at your images path access is being denied. Any chance you have a folder set to "Read Only"?

Well, I have scraped before with no issues.  And I can go into those folders and create and edit files, so, i don't think it's read only. 

I will try on a different computer to see if i get similar results although there is a significantly less number of games on that one.

[Retro808]

19 minutes ago, HomerJ said:

OK, this is the Noobs forum.    How do I import metadata for just one platform?

 

Just click the platform and then select all the games or how ever many you want. 
 

FYI you can also run the option to audit the platform and then select games from there. Just to give you another way of doing it. 

[HomerJ]

Well, I figured out the 'accesses denied issue'.  My noob fault on that one.

I thought I was being 'creative' and 'safe' by creating an admin account and then a BigBox account on windows.  Where I would do all the configuration and setup on the admin account, but play the games ont eh BB account.  

Well, appears I must have scraped some stuff on the BB account and it was unable to mess with the admin account files and vice versa.

So, I went and changed ownership of all the LB files to the admin and rescraping now.

I'm guessing that this might be causing the download stalling too.   Fingers crossed.

 

[HomerJ]

Didn't help.

Fixing the permissions, did allow all the metadata to be downloaded ok, but when it transitions to the media files, no go.  It stalls after just a few files.  When I cancel, quit, and restart, LB then instantly goes back to the beginning and redownloads all the meta data again....well over an hour process.

21 hours ago, Retro808 said:

Just click the platform and then select all the games or how ever many you want. 
 

FYI you can also run the option to audit the platform and then select games from there. Just to give you another way of doing it. 

Forgive me for my lack of knowledge here, but I still can't figure this out.  I select a platform, then I select the games....then what?  If I go tools->download metadata and media, it brings up the wizard and does everything.  How do i just do 1 platform?  I wonder if maybe, in general, this method works, but my LB thinks it's still in the middle of a larger operation because I had to cancel the import.  Is there a way to clean things out (without a clean install of LB)?

thanks for the help.

[Retro808]

22 minutes ago, HomerJ said:

Didn't help.

Fixing the permissions, did allow all the metadata to be downloaded ok, but when it transitions to the media files, no go.  It stalls after just a few files.  When I cancel, quit, and restart, LB then instantly goes back to the beginning and redownloads all the meta data again....well over an hour process.

It could be EmuMovies if you are also scraping media from them. We know there is an API issue causing some download issues. If you uncheck the media options and do not scrape for media from EmuMovies and just scrape from the Games DB does it give you the same issues?

24 minutes ago, HomerJ said:

Forgive me for my lack of knowledge here, but I still can't figure this out.  I select a platform, then I select the games....then what?  If I go tools->download metadata and media, it brings up the wizard and does everything.  How do i just do 1 platform?  I wonder if maybe, in general, this method works, but my LB thinks it's still in the middle of a larger operation because I had to cancel the import.  Is there a way to clean things out (without a clean install of LB)?

thanks for the help.

Once you highlight all the games you want to run media for you do go to >Tools >Download Media and Metadata. The screen will show you how many games you are trying to run media for. You can see I selected 2 games and the screen confirms 2 games in the text.

[HomerJ]

Again, thank you for you patients and help.....

That's not what's happening for me.

I have 4 games selected.  I choose Tool->Download Metadata and Media.  I get this:

 

I click all the way through and at the end I see:

 

When I click ok, it will update the metadata for everything and then try to download al media (unsuccessfully).

 

I am running 11.17

[GrantBillington]

I've been scraping for a couple of days now and only have some art for just a couple of hundred games... I am trying to do multiple systems at once so I might try doing it in batches.

[GrantBillington]

9 hours ago, Retro808 said:

Once you highlight all the games you want to run media for you do go to >Tools >Download Media and Metadata. The screen will show you how many games you are trying to run media for. You can see I selected 2 games and the screen confirms 2 games in the text.

This worked much better for me. I'm getting a file per 10 seconds now which is a great improvement over the file per ~10 minutes from doing a global download.

However I never got any confirmation on how many games I selected though. I did look for it but it still eluded me.

[HomerJ]

Still have issues.

I started from scratch.  Deleted everything.  One platform at a time.  It's been goin pretty well.  Now, it's back to the same thing.  It will scrape 10 files or so and then pause for many many minutes (like 10 or more) and then do a few more. 

Now, when it pauses I am getting the following error:

Is there any log or any way I can troubleshoot this?  It's getting really frustrating.

[HomerJ]

20 minutes later....1 media file scraped....and more errors.  Is this a server error?  I have screped 10s of thousands of files in the last few days.  Now all of a sudden, dead stopped.

 

And also, login to what?  How do I fix this error?  Where do I login?

 

Edited September 27, 2021 by HomerJ

[HomerJ]

Any thoughts?

So, today, it's still stalled out.  So I canceled it and then did a tools->download metadata and media.  But this time, i disabled all the EmuMovies check boxes.  It's currently working.

So, this is certainly related to EMuMovies.  I want movie clips....how do I go about troubleshooting this?  I have a lifetime membership to EmuMovies and i can log into it via chrome with no issues and my account states it has no restictions.

 

[Retro808]

33 minutes ago, HomerJ said:

Any thoughts?

So, today, it's still stalled out.  So I canceled it and then did a tools->download metadata and media.  But this time, i disabled all the EmuMovies check boxes.  It's currently working.

So, this is certainly related to EMuMovies.  I want movie clips....how do I go about troubleshooting this?  I have a lifetime membership to EmuMovies and i can log into it via chrome with no issues and my account states it has no restictions.

 

This has been addressed in the beta thread. Newest beta has a work around that seems to be helping some users. So you can either test the beta and see if it helps or wait for the next stable release (ETA currently unknown).