LaunchBox RAM Usage

[faviann]

Hi I've been having this issue for a while but I didn't care before because I was mainly using bigbox until recently.

Not sure what's either causing it or how to fix it. Simply put, whenever I start launchbox, for a minimum of 4 minutes launchbox keeps my 3600x at a usage very close to 100% while also filling my memory to almost 95%. After it reaches that point having everything go in a a swap file). 

When it finally idles If I look at the windows resource monitor, it has around 16GB of commited memory. Here's a picture showing the physical memory usage of Launchbox (along with probably the record of lowest memory usage for chrome):

Though not a problem exactly (I'm at 16GB of system RAM) it's a bother that everything else gets thrown in the swap file because launchbox usages way too much RAM. I'm wondering what's causing this insanely high amount of data. Is it a bug? if not, is it like caching all the images (and also videos + sound) it can until some threshold of system RAM usage is reached? Is there a way to limit the RAM usage.

NOTE: Physical RAM usage kindof becomes what I would expect (around 1.5GB) after 20 minutes of idling. Thing is.... commited memory is still at 16.2GB (swap + physical memory usage).

Here's what I tried to fix/understand my issue where I currently am. Made a backup of my :
1) Reduced the numbers of games from the +5000 to ~210.  ==> Stabilizes the memory allocation (and the insane CPU usage) quicker; around 4 minutes now.

2) Delete all other videos/images/sounds from other platforms. I didn't expect it to do anything but had to test. Did nothing.

3) Flushed all folder's content that would have cache as a name. No change.

4) Delete the metadata.xml file (180MB) in case that would be the issue. No change. 

5) Turned on the debug logs to check what happens. Turns out it's not very useful since I think it's mainly to track what the user does/clicks in the app.  Still I'll attach it to the post.

So I'm not sure what to do but since the playlist feature is not exactly available in BigBox (want to order games by release date) I'm stuck using launchbox but stuck is kind of the good word since it really slows to a crawl everytime I launch the app. I was about to use ILSpy to try and debug or at least track what's causing this insane amount of memory allocation but I thought I'd post here first. You guys definitely have more of a chance to point me to what stupid stuff I did that causes the app to behave this way.

Thanks for your help in advance.

EDIT #1: On a first look, it seems to be related to the GaussianBlur class. Not sure if it's of any use but I'd gladly disable whatever feature is causing this if somebody could kindly point me to where I should go.

Debug 2020-08-20 05-22-29 PM.log

Edited August 20, 2020 by faviann
Added suspected cause in an edit

[Retro808]

In LB >Tools>Options>Visuals>Backgrounds and turn down the background blur. You can also try using the "Old Default" theme since that one did not have the blur.

What version LB are you on?

[faviann]

I'm on 11.2 which seems to be the latest non-beta available. I thought the debug logs had that info, my bad.

Turns out you were right by the way. Lowering the background blur to zero seems to fix it. I have less than 1GB in use after launch and loading. I must say though, I find it unfortunate since having the blurred backgrounds was pleasing from an aesthetic perspective. 

I'm wondering if I'd have another config that's causing the gaussian blur to be "re-processed" everytime the app launches. If not, I'm guessing it's making the filter asynchronously and caching the blurred image in RAM explaining the ballooning CPU and memory usage on each startup. I'm curious, since I had issues with even "only" 217 games in my library, any idea who this feature is targeted for since it was on by default I think?

In the meantime, thanks for clearing that out and helping me out.

[faviann]

Bumping to ask if there is another spot in the forums to ask the question on the underlying bug.

The feature is really nice and would like to keep it working and active. I could give remote access, run a debug builds, configure for remote debugging, etc. How do I ping a dev for it?

Or even better if I can get to fix it myself. (it's only wishful thinking)

Edited September 2, 2020 by faviann

[C-Beats]

Last time I remember hearing about a user experiencing that much issue with the blur it was related to their graphic card and it's driver, if you haven't already try updated your video driver and see if that helps you. Also are you still on 11.2 or are you on the latest now? We use .NET Core for 11.3 and newer which uses slightly different dependencies so it may be possible that that version runs differently (for better or worse) on your system.

[faviann]

I updated drivers. I am also on the latest non-beta version (11.4 and can confirm, it's using .Net Core)

I also tried with a different GPU. I was on a 1080Ti, ran DDU to uninstall the drivers completely/cleanly and swapped to a Vega 64. Same issue. Removed the drivers again using DDU and reinstalled the 1080Ti.

From a code/programming perspective, when I'm activating the blur background (something other than 0), it's actually spamming the creation of the class GaussianBlur like crazy leading to the insane amount of memory allocation. Now as to why that is... code is obfuscated and reverse engineering why this happen would take way too long.

Is there any way I can help/assist solving the issue?

Edit: Also disabled the Windows 10 Virus Real time protection, just in case. Not that it would be related but worth mentioning that this possibility has been tested already.
Edit2: This post highlights the same issue it seems. It seems the regression could be coming from that version.

 

Edited September 3, 2020 by faviann

[faviann]

I'm close to the point of reinstalling the OS (Windows 10) but I know it would be useless considering the design of launchbox (portable app depending on little outside of its installed folder). Before attempting a useless OS reinstall, I wanted to debug one last time.  After some debugging, I know for sure it has to do with the gaussian blur working overtime, the question is why. I can confirm it seems completely unrelated to drivers since the blurring process is done through regular .net libs. At least my impression.

I'm a just bit frustrated since BigBox doesn't seem to have playlist support (at least I couldn't find it) and LaunchBox is unsusable when blur is active. I'm willing to get my hands dirty and make it easy/simple for a dev to find the bug (and maybe fix after should it be an easy one to fix). 

Point being, am I posting in the right area? Is there a discord, bug tracker, support mail address or something simlilar to reach/poke any on people on the dev team? If not, who should I poke? The reason is I feel it's not gonna get fixed if I just wait for someone to notice this.

PS: From experience, bugs like these are usually related to an unexpected config on the user's side and I'm desperately trying to see what I did that could be unexpected.

NOTES FOR DEVS: I can see a lot of threads with C# tasks from .NET TPL trying to apply some gaussian blur to bitmap data. And as a note, the gaussian blur runs for so long it's not hard to pint point or get a debugger in the right place. I'm sure with proper debugging symbols this would be super easy to find&fix.

[Retro808]

You are in the right spot and you do not have to be anywhere else poking. We have a discord but posting there means it will get lost easily in the rapid communication done in Discord. This thread will be your best avenue. One of the devs did reply and chances are good he will check up on this thread.

Thing is this is a two man dev team working on a lot of items for this program. Plus daily family needs. The recent update of LB to .Net Core 3.1 has had them tending to a good bit of fixes. Since there is a workaround in place by reducing or turning off blur this will likely be something that could take a back seat to more pressing items that need to be fixed. Not telling you to give up, just asking you follow-up every now and then here and know it is not a huge dev team. So it might take some time.

[faviann]

I did not realize that:
A)  It was such a small team
B) That a dev answered

Kudos for them holding the fort, mighty impressive I should say. Considering the amount of stuff it has going for it. 

On a more positive note I have a pretty good lead now.
I just did a "fresh install" of Launchbox and the bug still triggers immediately on first launch (insane memory alloc + CPU at 100% + machine lag). There's definitely something on my machine that triggers the insane workload. That's actually really good news. It pretty much confirms my theory where an OS config I have is the source of the issue. If I can't solve it quickly via reverse engineering I'll just reinstall Windows 10.

The big question that could really helpme figure this issue: What gets enqueud for blurring even on first launch of a fresh install of Launchbox? 

Edited September 4, 2020 by faviann

[neil9000]

Just now, faviann said:

I did not realize that:
A)  It was such a small team
B) That a dev answered

Kudos for them holding the fort, mighty impressive I should say. Considering the amount of stuff it has going for it. 

On a more positive note I have a pretty good lead now.
I just did a "fresh install" of Launchbox and the bug still triggers immediately on first launch (insane memory alloc + CPU at 100% + machine lag). There's definitely something on my machine that triggers the insane workload. That's actually really good news. It pretty much confirms my theory where an OS config I have is the source of the issue. If I can't solve it quickly via reverse engineering I'll just reinstall Windows 10.

The big question that could really helpme figure this issue: What gets enqueud for blurring even on first launch of a fresh install of Launchbox? 

Yeah this is a really strange one as i have never seen it anything like that, and i update every single beta.

This has been open about twelve hours at this point.

C-Beats and Jason Carr are the only devs here, and for the record, all moderators are volunteers, so we reply when we can, but we all have lives/jobs/family's, and we squeeze this in when we can. 

[Retro808]

@faviann Do you also have the option for Dynamic Box sizing on? If so try turning that off and see if anything changes. It should be in >Tools>Options>Boxes

Just something else for you to look at in your digging around.

 

[faviann]

 

32 minutes ago, Retro808 said:

@faviann Do you also have the option for Dynamic Box sizing on? If so try turning that off and see if anything changes. It should be in >Tools>Options>Boxes

Just something else for you to look at in your digging around.

 

Tried it but it turns out it did not fix the issue but I have something better.

I should say I have real good news. I found the culprit. You're not gonna get the answer without the story. I'll feel less like an idiot this way.

So I sat down and actually started reverse engineering the Gaussian Blur async state machine. So I wanted to figure what the hell is going on, what is the class doing, why it needs so much memory and understand why is the Process method so compute heavy.

After a while analyzing the class' state machine I realize there's the equivalent of these lines:

//One dimensional representation of a loaded bitmap.
var buffer1 = new int[bitmapHeight * bitmapWidth]
var buffer2 = new int[bitmapHeight * bitmapWidth]
var buffer3 = new int[bitmapHeight * bitmapWidth]
var buffer4 = new int[bitmapHeight * bitmapWidth]
...

Why is it important? Well turns out the pixel buffers were big... and by big I mean ENORMOUSLY BIG. We're talking something around 1.31GB for each individual array. So this now makes sense of why my memory allocation was high like crazy. It also, explained why the blur was taking so long to process. For the blur, It had to process arrays with a length of 351,550,000. So it made sense the machine slowed to a crawl, memory bandwith was saturated. That's how it ended up having a 16GB of commited memory. And since the GaussianBlur object (and its internal buffers) was reference in memory by another object (to be displayed later on I'm guessing), the .NET garbage collector could not collect any of it.

So with that said, while the blur code is more of a collateral victim than the the root cause of the bug. The poor blur code was only doing what we asked of him. The real question is not why is blurring so slow but instead why the hell am I sending 1.3GB of data to the Gaussian blur class. So I went back to disassembly to find the root cause of all this. 

I started looking at how blur operations were queued. Turns out there's an ImageLoader that does the job of asynchronously loading an image (from file in our case) and then queuing the blurring. So the obvious question, who was asking the ImageLoader to load an image (and blur it)? Especially considering my library was completely empty and it was on a fresh run of Launchbox. This was making no sense to me until I realized who was asking for an image to load. The MainViewModel! It's basically the class that well.... has all the data of the main windows of launchbox. It needs a background for the Launchbox main windows and it loaded my desktop wallpaper (default behavior on Launchbox). For those curious, it uses the following file for me: `%USERPROFILE%\AppData\Local\Packages\Microsoft.Windows.Photos_8wekyb3d8bbwe\LocalState\PhotosAppBackground`

So I went to see the file and make sense on how the hell it was forcing launchbox to load 1.31GB of data. At this moment, I realized how stupid this whole situation is.

Almost 3 years ago, I had finished setting up my HTPC & 4K TV and decided it was time for me to also have a kickass wallpaper. I found on a forums where people shared high resolution artwork/wallpapers of the games. I fell in love with the Shadow of the Colossus one. I download the image, set it as my background and continue my life, blissful, happy and satisfied of my new setup.... until today. The resolution of that god damn wallpaper was 25000 X 14060. And no, there's no typo or one too many zeroes. The wallpaper was 140MB. Once decompressed, it's what lead to the dreaded 1.31GB ?‍♂️. Yeah..... nothing to add here


So what did we learn? Users are "idiots", they will throw the most outrageous kind of data/input at you and expect you to deal with it without flinching. Jokes aside, in all my years programming I never thought I'd be in the situation where I'm one of those extreme user who had I really funky/weird/unrealistic non-sensical setup that causes a bug. I actually resized the image back to 4K.

On a more serious note, the conclusion is: always sanitize user input/data because you never know what kind of ridiculous user you may end up with (a.k.a. me). When images that are too big, they could be ignored, resized (in memory), etc. Anyways I'll leave you to decide if, when and how to solve this issue.

[Ble3azy]

at last someone gets it. I was complaining of extraordinary ram usage from just selecting games, and everyone was like no idea what your talking about.

thanks so much for this

[faviann]

On 9/6/2020 at 9:49 PM, Ble3azy said:

at last someone gets it. I was complaining of extraordinary ram usage from just selecting games, and everyone was like no idea what your talking about.

thanks so much for this

Hey it's a pleasure too. In hindsight the bug is obvious (like a big chunk of bugs). It's this golden rule you really learn through time. If there's any form of "user-provided" data, it needs to be validated or else there's a big risk of a bug hiding there. I've been burnt by this I don't know how many times now. Anyways, I was lucky for one thing; code was in .NET so I could use simple tools to debug the thing and identify the bug (even with the obfuscation).

It really makes me happy it could help somebody else. Now go enjoy launchbox!

[Jason Carr]

@faviann Thanks for the run-through. Unfortunately even just loading and resizing a 25,000 X 14,060 pixel image is probably going to be a serious performance hit, but I would guess that it's gonna be marginally quicker than blurring it. I'll do some testing and see what I can do.

[faviann]

@Jason Carr Agreed. To be fair, I was filling my 16GB of RAM to 100% which forced the OS to do some heavy memory swapping. That last bit exacerbated the issue way more than it should've (slugishness of the OS + CPU usage at max). So the reality is my use case was literally just that: extreme.

That's why I try to refrain from offering any form of definitive "solution". There's other constraints you have to deal with that I have no clue of. That's without mentioning the matter of where you'll have to insert that code without it being hacky/incurring tech debt. Point being, I've had the easier job of just having to find and point out the issue, not fixing it.

So while yes it's a bug/issue, I feel it's less urgent. We have a good workaround (resize) and it's not requiring disabling the feature.
Thanks again for the support.

PS: Should you ever want the background to test at one point here's a link for the wallpaper

[Jason Carr]

For the 11.5-beta-3 release, I've added some code in to resize images that are larger than the screen resolution before attempting to blur them. It does seem to significantly help performance with excessively large images.

[faviann]

7 hours ago, Jason Carr said:

For the 11.5-beta-3 release, I've added some code in to resize images that are larger than the screen resolution before attempting to blur them. It does seem to significantly help performance with excessively large images.

Great news, thanks again for handling the fix so quickly. Happy to hear it went well

[floodie]

guys do the actions and fixes in this thread only apply to Launch box? What about when in bigbox mode?

[Jason Carr]

8 hours ago, floodie said:

guys do the actions and fixes in this thread only apply to Launch box? What about when in bigbox mode?

The performance issue was specifically with blurring images, which only happens in LaunchBox, so the issue that was resolved does not apply to Big Box at all.