Duplicate images and LaunchBox 7.10-beta-2

[Pacman56]

Good morning @Jason Carr

Yesterday I downloaded 7.10-beta-2 and started beta testing the new duplicate image prevention functionality.

Prior to start the download metadata and media from within LaunchBox, I made sure that no duplicate images were present in my Images folder.
After running the download metadata and media against my "Amstrad CPC" Platform using the second option (Yes, but do not replace any existing fields or media),  a considerable amount of duplicate images (2040 images) were created in the "LaunchBox\Images\Amstrad CPC" folder.

Here is a text file containing the list of these duplicates with their CRC32.

Amstrad CPC (3).txt

I will do more in-depth testing for this duplicate images problem with that same platform by re-doing these test while isolating if these duplicates come from the LaunchBox games database or Emumovies or both and post my results.

 

[Pacman56]

I just finished re-testing the "download metadata and media" while isolating if these duplicates come from the LaunchBox games database or Emumovies.

Here are the results:

1. LaunchBox Database Only: When running "download metadata and media" and excluding Emumovies from the download process, no duplicate images are downloaded.
2. Emumovies Only: When running "download metadata and media" and excluding the LaunchBox Games database checkboxes from the download process, 1020 images (which already exist on my system) will be re-downloaded. 

The good news is that the CRC32 method you created on the games database is doing job perfectly  but there appear to be a problem with that method on the Emumovies API which is probably out of your control.

Here is a screenshot with hash information of one of these 1020 duplicate files.

[cammelspit]

@Jason Carr Just got Beta 3 and I must say... Holy hell in a handbasket is it SO SO SOOOOO much faster on startup. Not quite as fast as yours on the stream but it's very close so a BIG win there. The only oddity I can find is that when I select a platform category it makes the confirmation chime twice, once when I first hit the button and a second time when it actually switches to the category. Otherwise, the selection is fast and responsive.

It also seems to be switching views WAY faster, essentially instantly now. Was that intentional or just a happy side effect? I. I LOVE IT!

Edited May 1, 2017 by cammelspit

[cammelspit]

The moral of the story is to not switch categories too fast yet. :P

Object reference not set to an instance of an object.

App:     Big Box
Version: 7.10-beta-3
Type:    System.NullReferenceException
Site:    Boolean (Unbroken.LaunchBox.Wpf.BigBox.ViewModels.FiltersViewModelBase, Unbroken.LaunchBox.Data.Game)
Source:  BigBox

   at (FiltersViewModelBase , Game )
   at Unbroken.LaunchBox.Wpf.BigBox.ViewModels.FiltersViewModelBase.<selectionChangedTimer_Elapsed>b__153_3(Game game)
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at (Platform , IEnumerable`1 )
   at Unbroken.LaunchBox.Wpf.BigBox.ViewModels.FilterDetailsViewModel.GetDetails(Platform platform, IEnumerable`1 g)
   at Unbroken.LaunchBox.Wpf.BigBox.ViewModels.FilterDetailsViewModel..ctor(Platform platform, IEnumerable`1 games)
   at (FiltersViewModelBase )
   at Unbroken.LaunchBox.Wpf.BigBox.ViewModels.FiltersViewModelBase.<selectionChangedTimer_Elapsed>b__153_0()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Recent Log:

   1:07:56 PM VlcMediaPlayer.Dispose(bool disposing) HandleManager.Remove
   1:07:56 PM VlcMediaPlayer.Dispose(bool disposing) EventManager.Detach
   1:07:56 PM VlcMediaPlayer.Dispose(bool disposing) Free
   1:07:56 PM VlcMediaPlayer.Dispose(bool disposing) ReleaseMediaPlayerFunction
   1:07:56 PM VlcMediaPlayer.Dispose(bool disposing) HandleManager.Remove
   1:07:56 PM VlcMediaPlayer.Dispose(bool disposing) EventManager.Detach
   1:07:56 PM VlcMediaPlayer.Dispose(bool disposing) Free
   1:07:56 PM VlcMediaPlayer.Dispose(bool disposing) ReleaseMediaPlayerFunction
   1:07:58 PM Music.StopAndResumeBackground Start
   1:07:58 PM VlcMediaPlayer.Dispose(bool disposing) HandleManager.Remove
   1:07:58 PM VlcMediaPlayer.Dispose(bool disposing) EventManager.Detach
   1:07:58 PM VlcMediaPlayer.Dispose(bool disposing) Free
   1:07:58 PM VlcMediaPlayer.Dispose(bool disposing) ReleaseMediaPlayerFunction
   1:07:59 PM VlcMediaPlayer.Dispose(bool disposing) HandleManager.Remove
   1:07:59 PM VlcMediaPlayer.Dispose(bool disposing) EventManager.Detach
   1:07:59 PM VlcMediaPlayer.Dispose(bool disposing) Free
   1:07:59 PM VlcMediaPlayer.Dispose(bool disposing) ReleaseMediaPlayerFunction
   1:08:01 PM Music.StopAndResumeBackground Start
   1:08:04 PM Music.StopAndResumeBackground Start
   1:08:06 PM Music.Pause Start
   1:09:25 PM Music.Resume Start
   1:09:27 PM Music.StopAndResumeBackground Start
   1:09:27 PM VlcMediaPlayer.Dispose(bool disposing) HandleManager.Remove
   1:09:27 PM VlcMediaPlayer.Dispose(bool disposing) EventManager.Detach
   1:09:27 PM VlcMediaPlayer.Dispose(bool disposing) Free
   1:09:27 PM VlcMediaPlayer.Dispose(bool disposing) ReleaseMediaPlayerFunction
   1:09:28 PM VlcMediaPlayer.Dispose(bool disposing) HandleManager.Remove
   1:09:28 PM VlcMediaPlayer.Dispose(bool disposing) EventManager.Detach
   1:09:28 PM VlcMediaPlayer.Dispose(bool disposing) Free
   1:09:28 PM VlcMediaPlayer.Dispose(bool disposing) ReleaseMediaPlayerFunction
   1:09:29 PM Music.StopAndResumeBackground Start
   1:09:29 PM VlcMediaPlayer.Dispose(bool disposing) HandleManager.Remove
   1:09:29 PM VlcMediaPlayer.Dispose(bool disposing) EventManager.Detach
   1:09:29 PM VlcMediaPlayer.Dispose(bool disposing) Free
   1:09:29 PM VlcMediaPlayer.Dispose(bool disposing) ReleaseMediaPlayerFunction
   1:09:31 PM Music.StopAndResumeBackground Start
   1:09:31 PM VlcMediaPlayer.Dispose(bool disposing) HandleManager.Remove
   1:09:31 PM VlcMediaPlayer.Dispose(bool disposing) EventManager.Detach
   1:09:31 PM VlcMediaPlayer.Dispose(bool disposing) Free
   1:09:31 PM VlcMediaPlayer.Dispose(bool disposing) ReleaseMediaPlayerFunction
   1:09:32 PM VlcMediaPlayer.Dispose(bool disposing) HandleManager.Remove
   1:09:32 PM VlcMediaPlayer.Dispose(bool disposing) EventManager.Detach
   1:09:32 PM VlcMediaPlayer.Dispose(bool disposing) Free
   1:09:32 PM VlcMediaPlayer.Dispose(bool disposing) ReleaseMediaPlayerFunction
   1:10:11 PM Exception

 

[NJDave71]

You have cut the load time in half. Awesome.
 

But you might want to add a check to see if the startup video is still playing to allow it to complete before displaying UI or pressing a any Key to go right to the UI.

Glad i was able to watch the live stream today.  

[Pacman56]

@Jason Carr

I did a little more testing regarding the duplicate image prevention feature with 7.10-beta-3 and just ran across a situation where a duplicate image comes consistently from the LaunchBox games database rather than EmuMovies. 

Knowing that you have a lot on your plate, I'm gonna try to provide as much information as possible so that can easily reproduce the problem from your end. This instance of the problem is happening with the "Mattel Intellivision" platform.

Here's how you should be able to reproduce the problem:

1. Replace your "Mattel Intellivision.xml" with the one I provided below.
2. Start LaunchBox and select the game "Thunder Castle" from the games list
3. Start the "Download Metadata and Media..."
4. Accept all default but Exclude EmuMovies from the download process, the select "Yes, but do not replace any existing fields or media", then click Finish

Each time you repeat step 3-4, a new duplicate "Thunder Castle-0x.png" will be created as seen below.

Auditing Media...
Duplicate CRC32: [2ed327e6]
  File: [\\PC-ARCADE\LaunchBox\Images\Mattel Intellivision\Clear Logo\Thunder Castle-01.png]
  File: [\\PC-ARCADE\LaunchBox\Images\Mattel Intellivision\Clear Logo\Thunder Castle-02.png]
  File: [\\PC-ARCADE\LaunchBox\Images\Mattel Intellivision\Clear Logo\Thunder Castle-03.png]
 

Mattel Intellivision.xml

[Jason Carr]

Thanks @Pacman56; does that image get deleted when running the Clean Up Images process?

[Pacman56]

13 minutes ago, Jason Carr said:

Thanks @Pacman56; does that image get deleted when running the Clean Up Images process?

I'm running the Clean Up Images process now but this will take some time to complete.

Before running the clean up images, I backed-up my Entire LB folder, re-started LB and tried to reproduce the problem again to recreate those dups I had deleted myself.
For some unknown reason, duplicates are no longer created when updating metadata for that game. I am speechless.  lol

 

[The_Keeper86]

I got a random crash on beta 3.

 

https://pastebin.com/LWUCNeWN

[Jason Carr]

@Pacman56 Might it have something to do with using network paths? I haven't been able to replicate any issues as of yet.

@cammelspit I've added some debug code to help troubleshoot that error if it happens again, because I can't pinpoint it just yet. But thanks.

@The_Keeper86 Thanks; I believe I have that fixed for the next beta.

[emko]

@Jason Carr would it be possible for launchbox/big box to add a flag to the update installer to skip DirectX install? since we already know the user has installed Lunchbox and Big Box so they must have Direct X already installed? or is that not possible? that's the only time i have to bring out a keyboard to press yes/no on the direct x part and since i already have it i rather we just skip it.

 

What i mean is when Launchbox/Big Box triggers the update installation process can we not pass a flag like -nodirectx that installer will check and if its there skip the directx install part?

Edited May 2, 2017 by emko

[cammelspit]

Just wanted to mention how well the duplicate image feature is working. I saved 25GB and just about 60,000 images total. So far I can't tell if any other images have been removed erroneously but so far, seems to work well. I also haven't found any dupes that were not removed. I will let you know if I can find anything. Thank you very much for this one! Long time coming but worth the wait for sure!

[ninjagaidenguy]

Just wanted to post that the latest beta fixed the issues I was having with really slow launching of Platforms and general slowness. I can stop messing around with various fixes and start setting up my other systems. Thanks for all the work on this!

[Jason Carr]

@cammelspit @ninjagaidenguy Great to hear guys, thanks.

@emko We used to skip the DirectX install, but depending on what version you're coming from it is needed, and we've run into so many issues in the past with LaunchBox not working because of missing DirectX DLLs. So unfortunately we have to keep it in there.

[d8thstar]

how can one get their dirty little hands on the beta?  i can't wait to try!

[DOS76]

Go to tools options updates and check the beta checkbox restart the program and you should be prompted to download the beta. Very easy.

[d8thstar]

of course it was that easy, thanks

[syntax_X]

From my experience the only way to prevent dupe images is:

Start fresh and download all metadata from only Emumovies.

Do an image clean. You should find a heap of dupes still. (Same if you only downloaded from Launchbox DB)

Now audit each system and arrange an image column descending, highlight all roms with 0 images and download metadata from the Launchbox DB for them for only that image type.
Repeat for each column, and each system.....

I did try using the cleanup tool ( and 2 others) on my Mame clear logos which took it down to 156mb, but after I used the above method the same folder is only 60mb.

 

One would think that the final option when importing images "Yes, but do not update games with existing metadata" (or along those lines) would still download images for empty fields, but it literally wont touch a game if it has any type of metadata attached to it.

The option "Yes, but do not replace any existing fields." only really works if you choose to download solely from LB or EM. If you choose both it will result in downloading the same image from LB and EM with different names (most of the time)

So I guess if you used my above method you could quickly scan for meta updates if you chose "Yes, but do not replace any existing fields." and made sure you were only downloading from your "primary"  server ( in my instance EM)

Then to check for LB updates you'd have to do the audit per platform per column thing....

 

Edited June 4, 2017 by syntax_X