vertigoelectric

21,132 titles Amstrad CPC Atari 2600 Atari 7800 Atari Jaguar Atari Lynx ColecoVision Magnavox Odyssey 2 Mattel Intellivision NEC TurboGrafx-16 Nintendo 64 Nintendo DS Nintendo Entertainment System Nintendo Game Boy Nintendo Game Boy Advance Nintendo Game Boy Color Nintendo GameCube Nintendo Switch Nintendo Virtual Boy Nintendo Wii Nintendo Wii U Sega 32X Sega CD Sega Game Gear Sega Genesis Sega Master System Sony Playstation Sony Playstation 2 Sony PSP Super Nintendo Entertainment System WonderSwan WonderSwan Color

That's exactly what I wanted. Thank you! My ImageQueue.xml file was 38mb, which is a pretty massive xml file.

Okay so now I have a bit of a problem. That error that I shared in my previous post pops up when I try to cancel the media download. Then LB closes. When I open it again, it restarts the download. I'm unable to clear it. What do I do now?

Installed the new beta and I'm still having issues with the media downloads for 21,132 games. First of all, from the time I started the "Download metadata and media" process, it took almost 4 hours just to get the metadata before even starting the media downloads. Once it started the media downloading, I began getting "EmuMovies 404 error, updating URLs" still. It got stuck on file 1 for about 10 minutes, then seemed to download 2-4 okay but now it's stuck on file 5. There are a total of 81,867 files to download so leaving it like this could take an ungodly amount of time. I'll have to cancel it. log file: https://zerobin.net/?e55de595db0012b0#VsqcfIdkEF3yzI3B5nzmiqTZd9O7yDJdq2BD5mwrKRg= UPDATE: Just noticed this error had popped up as well: Here's the text of the error: Specified argument was out of the range of valid values. (Parameter 'toExclusive') App: LaunchBox Version: 11.13-beta-3 Type: System.ArgumentOutOfRangeException Site: System.Collections.Concurrent.OrderablePartitioner`1[System.Tuple`2[System.Int32,System.Int32]] Create(Int32, Int32) Source: System.Collections.Concurrent at System.Collections.Concurrent.Partitioner.Create(Int32 fromInclusive, Int32 toExclusive) at Unbroken.LaunchBox.Windows.Threading.ForEach[T](IEnumerable`1 collection, Action`1 action) at Unbroken.LaunchBox.Windows.Processes.ImageDownloader.UpdateEmuMoviesDownloads(Func`1 cancel, Action`2 reportProgress, QueueItem currentQueueItem) at FacadeMock.InitFocusDic(Func`1 cancel, Action`2 reportProgress, QueueItem currentQueueItem, FacadeMock ) at Unbroken.LaunchBox.Windows.Processes.ImageDownloader.ProcessQueue(Func`1 isCancelled, String activeImageGroup, Action`3 onUpdateAction, Action`2 onGameImageDownloaded) at Unbroken.LaunchBox.Windows.Desktop.ImageDownloaderHelp.<>c__DisplayClass0_0.InsertMetaError() at Unbroken.LaunchBox.Windows.Threading.<>c__DisplayClass0_0.AssetMultipartInstance(Object ) at Unbroken.LaunchBox.Windows.Threading.<>c__DisplayClass0_0.IncludeTransactionItem() at System.Threading.ThreadHelper.ThreadStart_Context(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location where exception was thrown --- at System.Threading.ThreadHelper.ThreadStart() Recent Log: 5:01:39 PM Exception

One thing I have noticed that I thought was kind of strange is the speed in which the initial search for platform media occurs. Regardless of which games I select, the first thing LB does is search for media for the relevant platforms (not sure why it needs to do this every time, by the way). If I choose a lot of games, perhaps across multiple platforms, then the time it takes to search for the platform media PER FILE is much longer, which like I said seems strange. UPDATE: It seems the amount of time it takes to search for platform media is relative to the amount of games needing to be updated for that platform. I was just observing a scan, and noticed that platforms with a lot of games took longer to search for the platform media than those with fewer games. I wonder why that is. Maybe I'm misinterpreting what is actually being searched for. I'd post a screenshot for confirmation but it has already moved on to the metadata downloading step of the process.

After some testing and analyzing, I'm pretty sure I understand what the problem is (see my previous posts in this thread for context). It's not exactly the volume of items being updated, although that is part of the issue. It's the amount of time it takes to process the items when interacting with EmuMovies online. As far as I can tell, LB gathers the information about the media it needs and requests it from "DeX Games Media Download Service API" at http://api.gamesdbase.com/ (which I'm assuming is how EmuMovies provides its media). This download service creates a temporary URL for each download, which typically looks something like this: "http://api.gamesdbase.com/795e38c62fc638306332313136346129625f3634622c312d64ffddc9fd5a626b625561c477572f2936b6d01b7269781b7663c4196bc4756063196c60afc638322f24275fc466c86d6a245a5c711b6d6b72565ac51f2631196d6b7258.png" What I believe is happening is, when LB takes too long to parse and compile the information for a large volume of games and media, these temporary URLs are expiring before the LB actually uses them, so when LB finally gets around to it, we get errors like "The remote server returned an error: (404) Session has expired." Then, because the URLs are expired, we run into the message "EmuMovies 404 error, updating URLs" in the log file, and it's here that massive delays are caused because LB has to re-gather all of the information from all of the games again, and the cycle continues. Off the top of my head, I think this issue could probably be resolved in a couple of ways: The Media Download Service could extend the lifetime of the URLs LB could process the game data in "chunks" at a time, instead of all at once (like maybe process 100 games, and queue the rest in the background). Furthermore this could be a possible user-adjustable amount. Any thoughts?

I suppose it's possible. I'll check the errors in the logs and do some specific testing for that.

I've been updating the media for my games in "sessions" at a time instead of all at once. I gathered some information about those sessions in case it's helpful in any way. Some of them went just fine, but at least a couple I had to stop early because of too many errors, or errors such as "updating URLs" which caused excessive delays. PLATFORMS: ColecoVision Magnavox Odyssey 2 Mattel Intellivision NEC TurboGrafx-16 Nintendo 64 1100 games to update 9048 files to download STARTED: 2021-07-04 11:54:21 AM FINISHED: 2021-07-04 03:29:12 PM DURATION: ~4 HOURS NOTABLE ERRORS: 24 of "EmuMovies 404 error, updating URLs" 242 of "The remote server returned an error: (404) File Does Not Exist." SESSION LOG: https://zerobin.net/?742b13096e3a77a0#4C9LhZ+zFipgIjNVwnh08ILBaQxntfABcDxhN1NYmL8= =================================================== PLATFORM: Nintendo DS 821 games to update 1057 files to download STARTED: 2021-07-04 06:35:15 PM FINISHED: 2021-07-04 07:03:48 PM DURATION: 30 MINS NOTABLE ERRORS: 13 of "An error occurred invoking the method. The destination thread no longer exists." SESSION LOG: https://zerobin.net/?6368c5b68000c440#KX0+ghcW+5qz9NnIWV+UrULnGSf05Qv/K9VyZ1vqp5c= =================================================== PLATFORM: Nintendo Entertainment System 1687 games to update 9036 files to download STARTED: 2021-07-04 07:07:58 PM FINISHED: 2021-07-04 11:35:05 PM DURATION: ~4.5 HOURS NOTABLE ERRORS: NONE =================================================== PLATFORM: Nintendo Game Boy 1103 games to update 4432 files to download STARTED: 2021-07-05 04:29:13 AM FINISHED: 2021-07-05 06:07:21 AM DURATION: ~2 HOURS NOTABLE ERRORS: NONE =================================================== PLATFORM: Nintendo Game Boy Advance 1723 games to update 18031 files to download STARTED: 2021-07-05 08:39:08 FINISHED: 2021-07-05 03:43:56 PM DURATION: ~7 HOURS NOTABLE ERRORS: 74 of "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." 27 of "The operation was canceled." 3 of "The operation has timed out." 33 of "The remote server returned an error: (404) File Does Not Exist." 3 of "EmuMovies 404 error, updating URLs" SESSION LOG: https://zerobin.net/?83ce2b740a56f6c2#3AslovHUD7a79NrQWmIQUW+i/S2VXaTPawUqdNsVIU4= ===================================================

@C-Beats So far it only seems to happen when I choose a large number of games to update. I thought it was the number of files requested, but now I'm not so sure. I had a request of over 34,000 files download without errors while another request of 23,000 files exhibited the problem. I'll keep testing. Currently I'm downloading data for ~800 games (11,966 files) without issue.

Okay so I started another media update for my full library about the time of my last post (~5 hours ago). It does take a while to scan the games and grab the metadata first before downloading any media, and I didn't time that, but as of now, 5 hours after starting, it's only on "Downloading file 3 of 134577". I studied the log file and I have a theory. Take a look at this segment: 2021-07-03 01:12:39 PM Downloading "http://api.gamesdbase.com/ddc862c637c6352d312e63c864312f5e342d345e365d323437ffddc9ed5abe5a9d5561c47757c6292c6bcf1b61c47667cf621fc0.png" to "G:\Emulation\LaunchBox\Images\Nintendo Game Boy Color\Box - Front\10 Pin Bowling-01.png" 2021-07-03 01:12:39 PM FIRST CHANCE EXCEPTION: The remote server returned an error: (404) Session has expired. Please login.. at System.Net.HttpWebRequest.SendRequest() 2021-07-03 01:12:39 PM FIRST CHANCE EXCEPTION: The remote server returned an error: (404) Session has expired. Please login.. at System.Net.HttpWebRequest.SendRequest() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 2021-07-03 01:12:39 PM FIRST CHANCE EXCEPTION: The remote server returned an error: (404) Session has expired. Please login.. at System.Net.HttpWebRequest.GetResponse() 2021-07-03 01:12:39 PM FIRST CHANCE EXCEPTION: The remote server returned an error: (404) Session has expired. Please login.. at System.Net.HttpWebRequest.GetResponse() at System.Net.WebClient.GetWebResponse(WebRequest request) at System.Net.WebClient.DownloadBits(WebRequest request, Stream writeStream) 2021-07-03 01:12:39 PM FIRST CHANCE EXCEPTION: The remote server returned an error: (404) Session has expired. Please login.. at System.Net.HttpWebRequest.GetResponse() at System.Net.WebClient.GetWebResponse(WebRequest request) at System.Net.WebClient.DownloadBits(WebRequest request, Stream writeStream) at System.Net.WebClient.DownloadFile(Uri address, String fileName) 2021-07-03 01:12:39 PM EmuMovies 404 error, updating URLs It seems there's still a bug with the connection to EmuMovies content. That's fine, but look at the last line there. "updating URLs". In the log, there are lengthy gaps of time after this is written to the log. My theory is that every time the error occurs it is updating URLs for ALL of the media I'm requesting to download, which is why it's taking so long. I have a couple of things I want to try. I'll once again try downloading media for smaller groups of games. If/when the error occurs in this case, it shouldn't take it very long to "update URLs" for a smaller list of items. I'm also going to try omitting EmuMovies from the media request... see how that goes. UPDATE: So I chose to download media for 14 games. There were 167 files to download. All downloaded without errors. I think the problem is trying to request downloads for too many games/files in one session. I'll begin to try larger lists and see if I can figure out whereabout the "threshold" is for these errors to start appearing. UPDATE 2: This time I'm downloading media for a platform containing 2,575 games, requiring 34,478 files to download. It's been running for a few hours and it's downloading at a reasonable rate (probably about 1 file per second). At the time of typing it has downloaded ~19,800 files. Only 4 errors reported in the LB interface, none of which have to do with EmuMovies. In fact, the log file doesn't contain any notable errors; nothing about EmuMovies, sessions, or updating URLs. When this is done I'll try again with a larger download request queue. I'm curious to see where the breaking point is here.

I'm doing some testing/debugging to see if I can figure out what the issue is. I was trying to update the media for several thousands of games. Instead of selecting everything to download I only selected one platform with 70 games and it seemed to process each item more quickly. I did look at the log files and during the several hours I left it going this morning there were many instances of the following error: 2021-07-03 06:48:01 AM FIRST CHANCE EXCEPTION: The remote server returned an error: (404) Session has expired. Please login.. Before updating to the latest beta, I got a lot of these errors in LB, and I'm aware it is a known issue. They don't show in the LB interface any more but there were a lot in the log file. When testing with fewer games, I wasn't getting the "Session has expired" error, but I was getting this: 2021-07-03 08:04:45 AM FIRST CHANCE EXCEPTION: Out of memory. at System.Drawing.Image.FromFile(String filename, Boolean useEmbeddedColorManagement) 2021-07-03 08:04:45 AM EXCEPTION IGNORED: Out of memory. at System.Drawing.Image.FromFile(String filename, Boolean useEmbeddedColorManagement) at Unbroken.LaunchBox.Windows.Desktop.BoxImageCache.StartJmsPrinter(Object ) at Unbroken.LaunchBox.Windows.Desktop.BoxImageCache.InvokeReadableCallback(Object config) I'll continue poking at it to see what I can find.

For some reason, it's taking an excessively long time to download the media for games. After several hours it only downloaded about 50 items. Anybody else facing this issue? Maybe it's just me...

Hmmm, yes I suppose that would have been an option. It's not an issue any more, though. I don't really need the two installations sync'd after all. In fact, it's better if they are separate. Thanks, though. Appreciate it.