Archive Cache Manager

[fraganator]

*NOTE* The attached beta plugin is for LaunchBox 12.8 beta, and will not work in earlier LaunchBox versions.

With the latest LaunchBox 12.8 beta, there's a handy new feature which allows you to enable ROM extraction on a per platform basis. I've tested the current version of Archive Cache Manager (v2.0.9) with the 12.8 beta and the core functionality works, though the plugin is obviously unaware of the new per platform extraction setting. This means the plugin's "Select ROM in Archive" right-click menu will be available in cases where it should be disabled, plus a few minor setup / teardown steps which shouldn't be executed.

Attached is a new beta version of the plugin which should correctly handle LB's new per platform extraction setting, so if you're running the 12.8 beta please download this version. Once 12.8 is out of beta, I'll release an official version of the plugin. Thanks!

 

ArchiveCacheManager.v2.0.10.beta.zip

[Laizhensil]

On 9/17/2021 at 2:03 PM, gonzalogil said:

@fraganator Works awesome! Just what i was looking for, I really appreciate your work on this!

In the future, i would like you to consider it to work with non-zipped roms...so that, i can keep my library on my NAS, and on first launch copy it over to local SSD! 

This is something I'm looking for, too.  Would be an awesome feature.  Regardless, great work, thanks a million, Frag.

[fraganator]

6 hours ago, Laizhensil said:

This is something I'm looking for, too.  Would be an awesome feature.  Regardless, great work, thanks a million, Frag.

It should be doable, though might only be effective for single file ROMs (so not ROMs with multiple files, such as cue+bin). Would that be a problem?

I've raised an issue on github as a reminder

[Laizhensil]

On 2/20/2022 at 9:54 PM, fraganator said:

It should be doable, though might only be effective for single file ROMs (so not ROMs with multiple files, such as cue+bin). Would that be a problem?

I've raised an issue on github as a reminder

Hey, I'm not going to complain about ANYTHING when someone's developing solutions like this for free!  Whatever you can make happen is extremely appreciated, and I'd even be amiable to donations.  Please do whatever you think is best in whatever way you think is best.

[Nixx]

I want to modify this plugin a little to suit my needs (I have a json file who describe most of the contents of the hacked version of the rom within my 7z, and i want to display the hack description on the selection windows).
But i fail to compile it. (I'm a total noob with visual studio and C#)
At first, i had issue with System.Drawing.Common, if i put 4.7.2 for Plugin i've got this error :

(Sorry in french, idk how can i change the lang) :

Quote

1>------ Début de la génération : Projet : Plugin, Configuration : Release Any CPU ------
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2203,5): warning MSB3277: détection de conflits non résolus entre différentes versions de "System.Drawing.Common".
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2203,5): warning MSB3277: Conflit existant entre "System.Drawing.Common, Version=4.0.0.2, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" et "System.Drawing.Common, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51".
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2203,5): warning MSB3277:     "System.Drawing.Common, Version=4.0.0.2, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" a été sélectionné, car il était de type primaire et "System.Drawing.Common, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" ne l'était pas.
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2203,5): warning MSB3277:     Références qui dépendent de "System.Drawing.Common, Version=4.0.0.2, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" [C:\Users\Mehdi\Documents\archive-cache-manager\src\packages\System.Drawing.Common.4.7.2\lib\net461\System.Drawing.Common.dll].
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2203,5): warning MSB3277:         C:\Users\Mehdi\Documents\archive-cache-manager\src\packages\System.Drawing.Common.4.7.2\lib\net461\System.Drawing.Common.dll
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2203,5): warning MSB3277:           Item Include d'un fichier projet ayant généré la référence "C:\Users\Mehdi\Documents\archive-cache-manager\src\packages\System.Drawing.Common.4.7.2\lib\net461\System.Drawing.Common.dll".
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2203,5): warning MSB3277:             System.Drawing.Common, Version=4.0.0.2, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51, processorArchitecture=MSIL
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2203,5): warning MSB3277:     Références qui dépendent de "System.Drawing.Common, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" [].
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2203,5): warning MSB3277:         C:\Users\Mehdi\LaunchBox\Core\Unbroken.LaunchBox.Plugins.dll
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2203,5): warning MSB3277:           Item Include d'un fichier projet ayant généré la référence "C:\Users\Mehdi\LaunchBox\Core\Unbroken.LaunchBox.Plugins.dll".
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2203,5): warning MSB3277:             Unbroken.LaunchBox.Plugins, Version=1.0.0.0, Culture=neutral, PublicKeyToken=558191e84f56affc, processorArchitecture=MSIL
1>C:\Users\Mehdi\Documents\archive-cache-manager\src\Plugin\PluginUtils.cs(20,62,20,73): error CS1061: 'IEmulatorPlatform' ne contient pas de définition pour 'AutoExtract' et aucune méthode d'extension accessible 'AutoExtract' acceptant un premier argument de type 'IEmulatorPlatform' n'a été trouvée (une directive using ou une référence d'assembly est-elle manquante ?)
1>C:\Users\Mehdi\Documents\archive-cache-manager\src\Plugin\PluginUtils.cs(20,104,20,115): error CS1061: 'IEmulatorPlatform' ne contient pas de définition pour 'AutoExtract' et aucune méthode d'extension accessible 'AutoExtract' acceptant un premier argument de type 'IEmulatorPlatform' n'a été trouvée (une directive using ou une référence d'assembly est-elle manquante ?)

I i put the last version of System.Drawing.Common (6.0.0), i've got right of a lot of errors but i stiff got that :

Quote

L'opération de génération a démarré...
1>------ Début de la génération : Projet : Plugin, Configuration : Release Any CPU ------
1>C:\Users\Mehdi\Documents\archive-cache-manager\src\Plugin\PluginUtils.cs(20,62,20,73): error CS1061: 'IEmulatorPlatform' ne contient pas de définition pour 'AutoExtract' et aucune méthode d'extension accessible 'AutoExtract' acceptant un premier argument de type 'IEmulatorPlatform' n'a été trouvée (une directive using ou une référence d'assembly est-elle manquante ?)
1>C:\Users\Mehdi\Documents\archive-cache-manager\src\Plugin\PluginUtils.cs(20,104,20,115): error CS1061: 'IEmulatorPlatform' ne contient pas de définition pour 'AutoExtract' et aucune méthode d'extension accessible 'AutoExtract' acceptant un premier argument de type 'IEmulatorPlatform' n'a été trouvée (une directive using ou une référence d'assembly est-elle manquante ?)
========== Génération : 0 a réussi, 1 a échoué, 2 à jour, 0 a été ignoré ==========
 

I use the last version of Unbroken.Launchbox.Plugins.dll, maybe that's the issue ?

 

 

[fraganator]

Hi @Nixx,

That sounds like a cool addition. It looks like you managed to resolve the System.Drawing.Common issue. The other error messages relate to a new property only available in the 12.8 beta version of Unbroken.Launchbox.Plugins.dll, specifically IEmulatorPlatform.AutoExtract.

You can try updating LaunchBox to the latest beta, then use that version of the dll. Otherwise you can grab the code for version 2.0.9 of the plugin, which still uses an older version of the dll.

[fraganator]

Version 2.0.10 of the plugin is now available, with support for the new per platform ROM extraction setting in LaunchBox 12.8. Please note this release is compatible only with LaunchBox 12.8 and above. If you're using an older version of LaunchBox, please use v2.0.9 of the plugin.

For the next few releases I'll provide two download versions, one compatible with LaunchBox 12.8+, and one for older versions of LaunchBox.

[Nixx]

I've got an unexpected issue with a big 7z files. The extraction seems to fail.
This is an extreme use case, i have all my SNES Roms within 7z file. My Super Mario World.7z  contains 25214 files, 2GB compressed, 58GB uncompressed.

I tested manually using the 7z GUI on windows, i got no problem to extract a single rom from the 7z, it do it instantly.
So i used process hacker to take a look, and i saw that process : "C:\Users\Mehdi\LaunchBox\ThirdParty\7-Zip\7-zip.exe" x "C:\coffre\Hackset-SNES\Super Mario World.7z" "-oC:\Users\Mehdi\LaunchBox\ArchiveCache\Super Mario World.7z - 811C0203E668356F44A566CA8291D118" -y -aoa -bsp1

Now i wonder why does the plugin extract the whole 7z file instead of just the required one, even outside this whole "oversized 7z usecase", that seems like an extreme waste of resources to me.

 

Edited March 10, 2022 by Nixx

[fraganator]

2 hours ago, Nixx said:

I've got an unexpected issue with a big 7z files. The extraction seems to fail.
This is an extreme use case, i have all my SNES Roms within 7z file. My Super Mario World.7z  contains 25214 files, 2GB compressed, 58GB uncompressed.

I tested manually using the 7z GUI on windows, i got no problem to extract a single rom from the 7z, it do it instantly.
So i used process hacker to take a look, and i saw that process : "C:\Users\Mehdi\LaunchBox\ThirdParty\7-Zip\7-zip.exe" x "C:\coffre\Hackset-SNES\Super Mario World.7z" "-oC:\Users\Mehdi\LaunchBox\ArchiveCache\Super Mario World.7z - 811C0203E668356F44A566CA8291D118" -y -aoa -bsp1

Now i wonder why does the plugin extract the whole 7z file instead of just the required one, even outside this whole "oversized 7z usecase", that seems like an extreme waste of resources to me.

 

Hi @Nixx, thanks for the report. Are you launching the game from the "Select ROM In Archive" right-click menu?

The plugin assumes the entire contents of the zip/7z/rar file are required to play the game, and so will extract everything from the archive. The selected file is then used by LaunchBox to start the emulator. I can see in this use case extracting 58GB worth of files to play a single game is overkill.

Perhaps adding an Extract All checkbox to the "Select ROM In Archive" window to handle either extracting all files, or extracting and playing only a single file? I'll give some thought to the best way of handling this use case.

[Nixx]

Maybe that should be an option per system ?
I mean, even if my exemple is extreme, i'm probably far from the only one who use this kind of scenario with multiple version of the same rom within a 7z (the priority editor itself feature a pattern about GoodMerge file naming).
For this kind of usecase, extracting the whole content is an huge waste of resources' (both cpu/time and space used by the cache).

[fraganator]

I thought a per emulator / platform setting might make sense too. I've raised on issue on github to track it.

[fraganator]

Hello everyone!

Version 2.11 of the plugin is now available (download here). There are two download options depending on your version of LaunchBox:

ArchiveCacheManager.v2.11.zip - For LaunchBox 12.8 and above
ArchiveCacheManager.v2.11_LaunchBox12.7AndOlder.zip - For LaunchBox 12.7 and older

New features include:

• Multi-disc support and automatic M3U generation
  • Extract and cache all discs in a multi-disc game
  • Generate and use M3U if configured for the emulator \ platform
• Custom filename priority for all emulators \ platforms
• Option to automatically check for plugin updates
• Updated 7-Zip to version 21.07
• Minor bug fixes

The multi-disc feature has required changes to just about every code file in the cache manager core. I've tested the plugin as exhaustively as I can, but there may be some use cases that haven't been accounted for, or some bugs introduced. If you do find any issues or bugs, please leave a message here or on github and I'll try get them fixed asap.

Plans for the next release:

• Refine multi-disc support (per emulator \ platform setting, manually extract + launch multi-disc game from right-click menu)
• Option to extract only single file from archive
• Option to copy non-archive files to cache
• Update config window with individual emulator \ platform options (multi-disc, extract single file, copy non-zip, etc)
• Update config window to match LaunchBox theme

See below for a summary of the new features.

----------------------

Multi-disc support and automatic M3U generation
This release includes a major new feature - support for multi-disc games and automatic M3U generation!

To use the multi-disc cache feature, enable the Multi-disc Support option in the Archive Cache Manager config (it's enabled be default).

In the emulator settings make sure Extract ROMs is enabled, and Use M3U Playlists for Multiple Discs is enabled (if supported by the emulator).

The following actions occur when launching a multi-disc game:

• All disc archives from a multi-disc game will be extracted and added to the archive cache.
  • If a disc is already cached, only the missing discs are extracted.
  • The progress indicator on LaunchBox's startup screen indicates the extraction progress across all discs.
• M3U files will be generated (one per disc) with the filename based on either the game ID (this is LaunchBox's M3U naming convention), or the game's title + version.
  • Configure the M3U naming convention with the Use Game ID As M3U Filename option in the config window (it's enabled by default).
  • Note that the M3U filename may be used by emulators to name save files. Using the game ID guarantees the save game files will be unique, but can be difficult to manually manage.
• A list of cached file paths are written to the M3U files, one path per disc. The file path for each disc is selected based on the emulator \ platform file priority if it exists, or the new All \ All priority (details below). This ensures only cue, gdi, or other disc table of contents type files appear in the M3U file.
  • The paths in each disc's M3U file will be in priority order for that disc. So Disc 2 of a 3 disc game will list entries in order 2,1,3. This is the same convention used by LaunchBox when launching a disc through the Play Version menu.
• If the emulator \ platform supports M3U files (as configured in LaunchBox), the generated M3U file will be used when launching the game.
• Otherwise a single disc will be launched, and swapping between cached discs can be done manually in the emulator.

A future release will overhaul the config window, and provide more granular options on a per emulator \ platform basis, but for now the multi-disc support option is global.

Custom filename priority for all emulators / platforms
This release includes a new global priority for selecting files within an archive. This priority is listed as All \ All in the plugin config window, and applies to all emulators and platforms. The default for the global priority is mds, gdi, cue, eboot.bin, which mimics LaunchBox's internal priority when launching a file from an archive. Note that specific emulator \ platform priorities take precedence over the All \ All priority list.

The All \ All priority is the basis for the contents of the multi-disc M3U file generation. Be careful removing entries such as cue, unless a specific emulator \ platform is used to handle cue and similar file types.

Automatically check for plugin updates
The plugin now includes an option to automatically check for updates when starting LaunchBox. This is a simple version check of the latest release on github, and nothing is automatically downloaded or installed. When this version of the plugin is first loaded when starting LaunchBox, a message box will appear asking the user if they'd like to enable the update check. The option can also be changed at any time in the config window. If a new update is found a message box will appear shortly after LaunchBox is started, with the option to open the plugin's release page in a browser.

Edited March 25, 2022 by fraganator
Add download link

[Rogerooo]

Can we extract roms to a path based on platform? For instance, since Dolphin will load NGC and Wii games, I would like to be able to extract them to their respective platform folder in order to give them specific configurations within Retroarch, like setting shaders and core options based on Content Folder.

Something like ".\ArchiveCache\Nintendo GameCube\GAME.iso" would be helpful I think.

Using Launchbox's default 7z settings, all platform games are extracted to a ".\ThirdParty\7z\Temp" folder and I can't target Content Folder based settings in Retroarch that way as all roms belong to the same Temp folder.

[fraganator]

15 hours ago, Rogerooo said:

Can we extract roms to a path based on platform? For instance, since Dolphin will load NGC and Wii games, I would like to be able to extract them to their respective platform folder in order to give them specific configurations within Retroarch, like setting shaders and core options based on Content Folder.

Something like ".\ArchiveCache\Nintendo GameCube\GAME.iso" would be helpful I think.

Using Launchbox's default 7z settings, all platform games are extracted to a ".\ThirdParty\7z\Temp" folder and I can't target Content Folder based settings in Retroarch that way as all roms belong to the same Temp folder.

Hi @Rogerooo

Thanks for the suggestion, it's a good idea. It should be a fairly simple addition - I'll see if I can add it into the next release. It's being tracked as an issue on github.

[Rogerooo]

3 hours ago, fraganator said:

Hi @Rogerooo

Thanks for the suggestion, it's a good idea. It should be a fairly simple addition - I'll see if I can add it into the next release. It's being tracked as an issue on github.

Cool, thanks for looking into it and by the way, congrats on the plugin, I have my roms stored on the cloud and this is a great caching tool without the need to mess around with custom batch scripts! I'm not sure about the symlinks but I suppose it shouldn't be a problem.

If you don't mind I have another suggestion. Be able to customize the path with bindings (a bit like SelectedGame.Platform on themes for example), I'm not sure if they are exposed through the api but it might be useful for different use cases other than Platform.
So in practice, the temp folder could be specified something like ".\ArchiveCache\{SelectedGame.Platform}\".

Cheers

[fraganator]

7 hours ago, Rogerooo said:

Cool, thanks for looking into it and by the way, congrats on the plugin, I have my roms stored on the cloud and this is a great caching tool without the need to mess around with custom batch scripts! I'm not sure about the symlinks but I suppose it shouldn't be a problem.

If you don't mind I have another suggestion. Be able to customize the path with bindings (a bit like SelectedGame.Platform on themes for example), I'm not sure if they are exposed through the api but it might be useful for different use cases other than Platform.
So in practice, the temp folder could be specified something like ".\ArchiveCache\{SelectedGame.Platform}\".

Cheers

Thanks for the feedback, happy to know it's a useful plugin for you

I was thinking of having a few selectable path options in the initial implementation, such as Platform, Title, and Emulator. Additional selection options or custom bindings can be added in future if required.

[Rogerooo]

That works as well. I personally just need the platform subfolder in order to use it with Retroarch's Content Folder but users might need other folders for other scenarios. Customization is always great!

Are you thinking about replacing symlinks when another game is requested for that particular subfolder? Since the files need to be at the root level of the platform folder there might be conflicts with filenames, deleting all previous symlinks when a game is loaded might help in this regard.

[fraganator]

Hi everyone,

Version 2.12 of the plugin is now available (download here). This is another big update, and includes the following new features.

• Option to copy non-archive files to the cache (Thanks @gonzalogil, @Laizhensil)
• Support for extracting additional formats (Thanks @Nixx)
  • Option to extract chd to cue+bin using chdman
  • Option to extract rvz, wia, gcz to iso using DolphinTool
• Option to specify a launch folder for cached games (game platform, game title, or emulator title) (Thanks @Rogerooo)
  • Useful for managing common RetroArch settings
• Smart Extract option to only extract required files from an archive (Thanks @Nixx)
  • Useful for merged ROM sets
• Emulator selection when launching a file from "Select ROM In Archive" list (Thanks @Nixx)
• Option to bypass LaunchBox's check the ROM file exists when launching a game
  • Allows launching cached game immediately - no waiting for slow disk spin-up or network latency
  • Allows playing cached games 'offline' if NAS or cloud storage unavailable
• Config window and UI overhaul
• Minor bug fixes

Thanks again to everyone who contributed feature ideas and provided feedback.

The plugin release page has been updated with documentation on the new options and features. Please see below for additional notes on this release.

CHD and RVZ extraction
These new extraction options require placing a copy of chdman.exe and\or DolphinTool.exe in the Extractors subfolder of the ArchiveCacheManager plugin folder. These applications haven't been included with the plugin to keep the file size small, and because they won't be commonly used options.

Why have an option to extract CHD files? <Emulator> already supports them!
CHDs have become the ROM compression format of choice, and offer very good space savings. This option is for folks who want the smaller size of CHDs, while retaining the ability to play using emulators that don't have CHD support, or where CHDs may not run as well as the uncompressed disc image on their system. The majority of users won't want or need to touch this option.

That said, implementing it provided an opportunity to rework and modularize the backend extraction logic, so in theory any compression format can now be supported. If there's a particular format you'd like to see supported in a future release, let me know and it should be simple to add.

Caching non-archive files
The plugin now has the option to copy and cache any game file. When copying a game to the cache, only the file in the game's Application Path will be copied, so the game file needs to be able to run stand-alone. If the game relies on additional files (data files and so on) these won't be copied, and the game probably won't launch correctly.

[JamesBond@ge]

The UI is a little bit screwed up for me. The top of the UI is cut off 😞

This is from my TV at 1920x1080, running Windows 7, and I have 150% DPI settings / scale thingamabob to make the text on my screen bigger.

Edited April 24, 2022 by JamesBond@ge

[fraganator]

1 hour ago, JamesBond@ge said:

The UI is a little bit screwed up for me. The top of the UI is cut off 😞

This is from my TV at 1920x1080, running Windows 7, and I have 150% DPI settings / scale thingamabob to make the text on my screen bigger.

Thanks for the bug report, I was able to reproduce it on my machine.

I've created and attached a quick build which should make the config window fit on screen at 150% DPI. Could you test it out and let me know if it fits on your screen?

 

ArchiveCacheManager_v2.12_DPI.zip

Edited April 24, 2022 by fraganator