11.13 Beta Testing

[OhBoy]

There's still an issue with region tags being ignored on files with more than one region in the file name. IMO this is a critical bug, as a huge number of games don't get a region due to this. Also games from Taiwan don't get regions even when it's the only region in the file name. Import any Taiwan game and you'll see the region field will be empty. Same goes for any game with  multiple regions other than USA, Japan, or Europe. So "Crazy Taxi (USA, Japan).cdi" will import correctly, but "Crazy Taxi (USA, Brazil).cdi" will not import correctly, the region field will be blank.

[neil9000]

16 minutes ago, vertigoelectric said:

For some reason, it's taking an excessively long time to download the media for games.  After several hours it only downloaded about 50 items.  Anybody else facing this issue?  Maybe it's just me...

Ive just done a MAME fullset import about a hour ago that downloaded about 10,000 images, it took about 30 minutes.

[vertigoelectric]

9 minutes ago, neil9000 said:

Ive just done a MAME fullset import about a hour ago that downloaded about 10,000 images, it took about 30 minutes.

I'm doing some testing/debugging to see if I can figure out what the issue is.  I was trying to update the media for several thousands of games.  Instead of selecting everything to download I only selected one platform with 70 games and it seemed to process each item more quickly.  I did look at the log files and during the several hours I left it going this morning there were many instances of the following error:

2021-07-03 06:48:01 AM FIRST CHANCE EXCEPTION: The remote server returned an error: (404) Session has expired. Please login..

Before updating to the latest beta, I got a lot of these errors in LB, and I'm aware it is a known issue.  They don't show in the LB interface any more but there were a lot in the log file.

When testing with fewer games, I wasn't getting the "Session has expired" error, but I was getting this:

2021-07-03 08:04:45 AM FIRST CHANCE EXCEPTION: Out of memory.
   at System.Drawing.Image.FromFile(String filename, Boolean useEmbeddedColorManagement)
2021-07-03 08:04:45 AM EXCEPTION IGNORED: Out of memory.
   at System.Drawing.Image.FromFile(String filename, Boolean useEmbeddedColorManagement)
   at Unbroken.LaunchBox.Windows.Desktop.BoxImageCache.StartJmsPrinter(Object )
   at Unbroken.LaunchBox.Windows.Desktop.BoxImageCache.InvokeReadableCallback(Object config)

   

I'll continue poking at it to see what I can find.

Edited July 3, 2021 by vertigoelectric

[vertigoelectric]

Okay so I started another media update for my full library about the time of my last post (~5 hours ago).  It does take a while to scan the games and grab the metadata first before downloading any media, and I didn't time that, but as of now, 5 hours after starting, it's only on "Downloading file 3 of 134577".

I studied the log file and I have a theory.  Take a look at this segment:

2021-07-03 01:12:39 PM Downloading "http://api.gamesdbase.com/ddc862c637c6352d312e63c864312f5e342d345e365d323437ffddc9ed5abe5a9d5561c47757c6292c6bcf1b61c47667cf621fc0.png" to "G:\Emulation\LaunchBox\Images\Nintendo Game Boy Color\Box - Front\10 Pin Bowling-01.png"
2021-07-03 01:12:39 PM FIRST CHANCE EXCEPTION: The remote server returned an error: (404) Session has expired. Please login..
   at System.Net.HttpWebRequest.SendRequest()
2021-07-03 01:12:39 PM FIRST CHANCE EXCEPTION: The remote server returned an error: (404) Session has expired. Please login..
   at System.Net.HttpWebRequest.SendRequest()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2021-07-03 01:12:39 PM FIRST CHANCE EXCEPTION: The remote server returned an error: (404) Session has expired. Please login..
   at System.Net.HttpWebRequest.GetResponse()
2021-07-03 01:12:39 PM FIRST CHANCE EXCEPTION: The remote server returned an error: (404) Session has expired. Please login..
   at System.Net.HttpWebRequest.GetResponse()
   at System.Net.WebClient.GetWebResponse(WebRequest request)
   at System.Net.WebClient.DownloadBits(WebRequest request, Stream writeStream)
2021-07-03 01:12:39 PM FIRST CHANCE EXCEPTION: The remote server returned an error: (404) Session has expired. Please login..
   at System.Net.HttpWebRequest.GetResponse()
   at System.Net.WebClient.GetWebResponse(WebRequest request)
   at System.Net.WebClient.DownloadBits(WebRequest request, Stream writeStream)
   at System.Net.WebClient.DownloadFile(Uri address, String fileName)
2021-07-03 01:12:39 PM EmuMovies 404 error, updating URLs

It seems there's still a bug with the connection to EmuMovies content.  That's fine, but look at the last line there.  "updating URLs".  In the log, there are lengthy gaps of time after this is written to the log.  My theory is that every time the error occurs it is updating URLs for ALL of the media I'm requesting to download, which is why it's taking so long.

I have a couple of things I want to try.  I'll once again try downloading media for smaller groups of games.  If/when the error occurs in this case, it shouldn't take it very long to "update URLs" for a smaller list of items.  I'm also going to try omitting EmuMovies from the media request... see how that goes.

 

UPDATE:

So I chose to download media for 14 games.  There were 167 files to download.  All downloaded without errors.  I think the problem is trying to request downloads for too many games/files in one session.  I'll begin to try larger lists and see if I can figure out whereabout the "threshold" is for these errors to start appearing.

 

UPDATE 2:

This time I'm downloading media for a platform containing 2,575 games, requiring 34,478 files to download.  It's been running for a few hours and it's downloading at a reasonable rate (probably about 1 file per second).  At the time of typing it has downloaded ~19,800 files.  Only 4 errors reported in the LB interface, none of which have to do with EmuMovies.  In fact, the log file doesn't contain any notable errors; nothing about EmuMovies, sessions, or updating URLs.  When this is done I'll try again with a larger download request queue.  I'm curious to see where the breaking point is here.  

Edited July 4, 2021 by vertigoelectric

[raverman72]

nice. want to try

[neil9000]

36 minutes ago, raverman72 said:

nice. want to try

Any actual comment or question related to the current betas? Or do i just immediately delete your comment, being it's the only one you have ever made?

[FistyDollars]

I'm having a strange issue - after importing my Steam library, some games don't have any Steam Metadata, or even Steam Banners/Posters. It seems like this only happens to ones that also didn't get matched up to the LaunchBox Games Database during importing. Additionally, when I go to download metadata for any of my Steam games (whether they matched or not) I'm not given an option to download Steam Banners/Posters.

[RULLUR]

Still not sure why I'm having times where all my Clear Logos aren't loading up like they should. I've tried everything from cleaning out Cache-BB and force populating them back as well. but still no improvements as I wait for each Beta to maybe resolve it.

Edited July 4, 2021 by C-Beats
Removed potentially drama causing rant

[C-Beats]

I saw the original post, then edited it as I'd like to keep this thread on topic. I believe Neil was simply trying provide the reporting user with data for comparison, that he is NOT having the issue and a time frame that the feature should complete in when working correctly. I don't believe it was intended as a "git gud internetz" sorta comment.

@vertigoelectric You are correct. We are attempting to correct URLS when you EmuMovies session expires. This should NOT be occurring after every download though (since the first time it needs to do it should update any session information or url). Let me know if you continue to have the issue over the weekend.

@OhBoy Our multi-region code is working but the way it is implemented doesn't register every possible combination. I believe I may be able to improve the code to help in situations like yours but that change has not made it into any builds yet. I can't promise it will make it into this release, but I'll do my best to get it addressed.

@RULLUR We are aware that you are having an issue loading images in your wheels. You've made it very clear to us over several threads. We have looked into the report and are having problems replicating it though to be able to track down any cause for it. I am also aware that unless you have changed how you organize your library that your set up is quite a bit different than most and so I'm curious if that isn't aggravating the issue for you. If you'd like PM me with any details you can about how often you are seeing this issue, if it is affecting one view more than others, or affecting one platform/playlist/category more than others and maybe we can deduce why you are seeing what you are.

[vertigoelectric]

22 minutes ago, C-Beats said:

@vertigoelectric You are correct. We are attempting to correct URLS when you EmuMovies session expires. This should NOT be occurring after every download though (since the first time it needs to do it should update any session information or url). Let me know if you continue to have the issue over the weekend.

@C-Beats So far it only seems to happen when I choose a large number of games to update.  I thought it was the number of files requested, but now I'm not so sure.  I had a request of over 34,000 files download without errors while another request of 23,000 files exhibited the problem.  I'll keep testing.  Currently I'm downloading data for ~800 games (11,966 files) without issue.

[RULLUR]

2 hours ago, C-Beats said:

I saw the original post, then edited it as I'd like to keep this thread on topic. I believe Neil was simply trying provide the reporting user with data for comparison, that he is NOT having the issue and a time frame that the feature should complete in when working correctly. I don't believe it was intended as a "git gud internetz" sorta comment.

@vertigoelectric You are correct. We are attempting to correct URLS when you EmuMovies session expires. This should NOT be occurring after every download though (since the first time it needs to do it should update any session information or url). Let me know if you continue to have the issue over the weekend.

@OhBoy Our multi-region code is working but the way it is implemented doesn't register every possible combination. I believe I may be able to improve the code to help in situations like yours but that change has not made it into any builds yet. I can't promise it will make it into this release, but I'll do my best to get it addressed.

@RULLUR We are aware that you are having an issue loading images in your wheels. You've made it very clear to us over several threads. We have looked into the report and are having problems replicating it though to be able to track down any cause for it. I am also aware that unless you have changed how you organize your library that your set up is quite a bit different than most and so I'm curious if that isn't aggravating the issue for you. If you'd like PM me with any details you can about how often you are seeing this issue, if it is affecting one view more than others, or affecting one platform/playlist/category more than others and maybe we can deduce why you are seeing what you are.

I mainly keep my stuff set as Default as I can for the most part without setting any Themes to it. I in a way feel that's best for Testing the base of things and not having to guess any issue I have might've been in result of a Theme being not 100% compatible with some changes in a Beta that comes along. I will PM anything at request to get to the bottom of this issue

Edited July 4, 2021 by C-Beats
Removing off topic rant... again

[C-Beats]

@RULLUR Please keep your responses focused on the beta. If you have grievances you wish to file about a mod please email support@unbrokensoftware.com.

[OhBoy]

On 7/4/2021 at 6:13 AM, C-Beats said:

@OhBoy Our multi-region code is working but the way it is implemented doesn't register every possible combination. I believe I may be able to improve the code to help in situations like yours but that change has not made it into any builds yet. I can't promise it will make it into this release, but I'll do my best to get it addressed.

Sounds good man, thank you. I also just experienced the bug where changing an image type causes the image to get deleted. For example if I change an image from Box Front Reconstructed to Box Front, that image get's deleted instead.

[vertigoelectric]

I've been updating the media for my games in "sessions" at a time instead of all at once.  I gathered some information about those sessions in case it's helpful in any way.  Some of them went just fine, but at least a couple I had to stop early because of too many errors, or errors such as "updating URLs" which caused excessive delays.

PLATFORMS:
	ColecoVision         
	Magnavox Odyssey 2   
	Mattel Intellivision 
	NEC TurboGrafx-16    
	Nintendo 64          
	

1100 games to update
9048 files to download

STARTED:  2021-07-04 11:54:21 AM
FINISHED: 2021-07-04 03:29:12 PM
DURATION: ~4 HOURS

NOTABLE ERRORS:
	24 of "EmuMovies 404 error, updating URLs"
	242 of "The remote server returned an error: (404) File Does Not Exist."
	
SESSION LOG:  https://zerobin.net/?742b13096e3a77a0#4C9LhZ+zFipgIjNVwnh08ILBaQxntfABcDxhN1NYmL8=
===================================================

PLATFORM:
	Nintendo DS

821 games to update
1057 files to download

STARTED:  2021-07-04 06:35:15 PM
FINISHED: 2021-07-04 07:03:48 PM
DURATION: 30 MINS

NOTABLE ERRORS:
	13 of "An error occurred invoking the method.  The destination thread no longer exists."

SESSION LOG: https://zerobin.net/?6368c5b68000c440#KX0+ghcW+5qz9NnIWV+UrULnGSf05Qv/K9VyZ1vqp5c=
===================================================

PLATFORM:
	Nintendo Entertainment System

1687 games to update
9036 files to download

STARTED:  2021-07-04 07:07:58 PM
FINISHED: 2021-07-04 11:35:05 PM
DURATION: ~4.5 HOURS

NOTABLE ERRORS: NONE
===================================================

PLATFORM:
	Nintendo Game Boy

1103 games to update
4432 files to download

STARTED:  2021-07-05 04:29:13 AM
FINISHED: 2021-07-05 06:07:21 AM
DURATION: ~2 HOURS

NOTABLE ERRORS: NONE
===================================================

PLATFORM:
	Nintendo Game Boy Advance

1723 games to update
18031 files to download

STARTED:  2021-07-05 08:39:08
FINISHED: 2021-07-05 03:43:56 PM
DURATION: ~7 HOURS

NOTABLE ERRORS: 
	74 of "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond."
	27 of "The operation was canceled."
	3  of "The operation has timed out."
	33 of "The remote server returned an error: (404) File Does Not Exist."
	3  of "EmuMovies 404 error, updating URLs"

SESSION LOG:  https://zerobin.net/?83ce2b740a56f6c2#3AslovHUD7a79NrQWmIQUW+i/S2VXaTPawUqdNsVIU4=
===================================================

 

Edited July 5, 2021 by vertigoelectric

[mcfilmmakers]

On 7/4/2021 at 9:41 AM, vertigoelectric said:

@C-Beats So far it only seems to happen when I choose a large number of games to update.  I thought it was the number of files requested, but now I'm not so sure.  I had a request of over 34,000 files download without errors while another request of 23,000 files exhibited the problem.  I'll keep testing.  Currently I'm downloading data for ~800 games (11,966 files) without issue.

Have you considered it might be a filename or even game name issue (ex: special characters?). Just a thought...

[vertigoelectric]

4 minutes ago, mcfilmmakers said:

Have you considered it might be a filename or even game name issue (ex: special characters?). Just a thought...

I suppose it's possible.  I'll check the errors in the logs and do some specific testing for that.

[mcfilmmakers]

42 minutes ago, vertigoelectric said:

I suppose it's possible.  I'll check the errors in the logs and do some specific testing for that.

I bring it up because it might be as simple as one specific game causing this issue. Try narrowing your selection down, rather than expanding out by separating your problem into chunks and testing each chunk rather than only looking at one chunk and ignoring the rest. Once you find the problem chunk, divide it again, and rinse and repeat.

If all chunks have no problem, then it would have to be a volume problem.

Edited July 6, 2021 by mcfilmmakers

[vertigoelectric]

After some testing and analyzing, I'm pretty sure I understand what the problem is (see my previous posts in this thread for context).  It's not exactly the volume of items being updated, although that is part of the issue.  It's the amount of time it takes to process the items when interacting with EmuMovies online.

As far as I can tell, LB gathers the information about the media it needs and requests it from "DeX Games Media Download Service API" at http://api.gamesdbase.com/ (which I'm assuming is how EmuMovies provides its media).  This download service creates a temporary URL for each download, which typically looks something like this:

"http://api.gamesdbase.com/795e38c62fc638306332313136346129625f3634622c312d64ffddc9fd5a626b625561c477572f2936b6d01b7269781b7663c4196bc4756063196c60afc638322f24275fc466c86d6a245a5c711b6d6b72565ac51f2631196d6b7258.png"

What I believe is happening is, when LB takes too long to parse and compile the information for a large volume of games and media, these temporary URLs are expiring before the LB actually uses them, so when LB finally gets around to it, we get errors like "The remote server returned an error: (404) Session has expired."  Then, because the URLs are expired, we run into the message "EmuMovies 404 error, updating URLs" in the log file, and it's here that massive delays are caused because LB has to re-gather all of the information from all of the games again, and the cycle continues.

Off the top of my head, I think this issue could probably be resolved in a couple of ways:

• The Media Download Service could extend the lifetime of the URLs
• LB could process the game data in "chunks" at a time, instead of all at once (like maybe process 100 games, and queue the rest in the background).  Furthermore this could be a possible user-adjustable amount.

 

Any thoughts?

[C-Beats]

You are accurately describing what is causing the 404. If you don't touch an EmuMovies link in 15m or so your session expires and we have to generate new links. The issue you personally are having in the process is that the "generate new links" step is taking an absurd amount of time and is actually taking LONGER than that 15 minute window meaning we are essentially doing it every download. I am working to see why that process is taking so long on your machine and seeing if there are ways to expedite that process.

[vertigoelectric]

56 minutes ago, C-Beats said:

You are accurately describing what is causing the 404. If you don't touch an EmuMovies link in 15m or so your session expires and we have to generate new links. The issue you personally are having in the process is that the "generate new links" step is taking an absurd amount of time and is actually taking LONGER than that 15 minute window meaning we are essentially doing it every download. I am working to see why that process is taking so long on your machine and seeing if there are ways to expedite that process.

One thing I have noticed that I thought was kind of strange is the speed in which the initial search for platform media occurs.  Regardless of which games I select, the first thing LB does is search for media for the relevant platforms (not sure why it needs to do this every time, by the way).  If I choose a lot of games, perhaps across multiple platforms, then the time it takes to search for the platform media PER FILE is much longer, which like I said seems strange.  

UPDATE:  It seems the amount of time it takes to search for platform media is relative to the amount of games needing to be updated for that platform.  I was just observing a scan, and noticed that platforms with a lot of games took longer to search for the platform media than those with fewer games.  I wonder why that is.  Maybe I'm misinterpreting what is actually being searched for.  I'd post a screenshot for confirmation but it has already moved on to the metadata downloading step of the process.

Edited July 6, 2021 by vertigoelectric