Archive Cache Manager

[fraganator]

On 9/8/2023 at 8:07 PM, W4rCh1ld said:

No its not, just on an External USB HDD... so i cannot exclude that folder from being sync, but i did try replacing the folder with a fresh one from a new install, using your program in a completely fresh install, and even replacing the ThirdParty\7-Zip with a full current 64bit version of 7zip (which works fine with lunchbox as is and infact did make all the extraction work far faster, than the included base\standard version) but this doesn't link to your Cache manager... it would be great to be able to use a full 7zip install in you cache manager and add it in ourselves, like CHDman, DolphinTool and Extractxiso as it seems to be a better version then the one lauchbox uses anyway, and should help to avoid issues with linking to the default one? Maybe?

Hi @W4rCh1ld - I believe I've found the issue. When a game is launched, Archive Cache Manager copies some files to the ThirdParty\7-Zip folder. Once the game has launched, the original 7-zip files are then restored. The issue in the logs showed those files were in use by another process. After a little debugging I think what is happening is LaunchBox is calculating RetroAchievement ID for the game in the background, which means the game is being extracted using 7-zip, and so the 7-zip files are in use.

I've attached a beta version which doesn't replace the 7-zip files on each launch, but instead replaces them once when LaunchBox is first launched. Any calls to 7-zip be LaunchBox should continue to operate as normal (unzipping metadata, creating start-up and shutdown backups, and RetroAchievement ID scans). Please let me know how it goes.

ArchiveCacheManager.v2.16.Beta1.zip

[W4rCh1ld]

17 hours ago, fraganator said:

Hi @W4rCh1ld - I believe I've found the issue. When a game is launched, Archive Cache Manager copies some files to the ThirdParty\7-Zip folder. Once the game has launched, the original 7-zip files are then restored. The issue in the logs showed those files were in use by another process. After a little debugging I think what is happening is LaunchBox is calculating RetroAchievement ID for the game in the background, which means the game is being extracted using 7-zip, and so the 7-zip files are in use.

I've attached a beta version which doesn't replace the 7-zip files on each launch, but instead replaces them once when LaunchBox is first launched. Any calls to 7-zip be LaunchBox should continue to operate as normal (unzipping metadata, creating start-up and shutdown backups, and RetroAchievement ID scans). Please let me know how it goes.

ArchiveCacheManager.v2.16.Beta1.zip 2.38 MB · 1 download

Well ive gotta say that works an absolute treat!! Unzipping file after file back to back, no issues, once id added in the emulator and specified the file i wanted in the list, picking the correct file in zips that have 2 files in (which launchbox is sadly so bad at it simply mounts the wrong file 100% of the time, not even with 50/50 odds of getting it right. lol), skipping the launchbox link check and booting pre-cached games 4x quicker, filling a full 20gig cache and auto deleting the oldest automatically, keeping the one id check to be kept, and deleting the next oldest. Beautiful! I haven't been able to check if its completing RetroAchievement ID scans correctly (although i bet your right that it was this causing the issue) as the emulator im using doesnt have the ability to use them and Retro Achievements are undergoing all sorts of issues and changes on their end that Launchbox is a mess in that regard atm. So yeah thank you very much for what is possible the most useful plug in ive seen for lauchbox (my other favourite bring PCSX2 Configurator Next, which is also an amazing tool) that they should really consider adopting it as standard. Also what is even better is now with this fix even when replacing the standard 7zip file within the 3rd party 7zip folder with a full 64x 7zip install, (where lauchbox unzips work faster) is your cache manager DOES recognize this 7zip .exe / File too!! (Where im pretty sure it didn't seem to before, at all.) So the best of both worlds, (you may even want to mention doing this in your Install instructions as an optional performace boost. ) .Now im super happy. Thank you so much for looking into it.

Further to this i have now made the Archive Cache folder path to an SSD with a 200gig max size, making pre-cached boot times of large files extremely fast... i can only imagine what a NVME or even a Ram Cache could achieve. lol

Edited September 10, 2023 by W4rCh1ld

[fraganator]

On 9/10/2023 at 1:46 AM, W4rCh1ld said:

Well ive gotta say that works an absolute treat!! Unzipping file after file back to back, no issues, once id added in the emulator and specified the file i wanted in the list, picking the correct file in zips that have 2 files in (which launchbox is sadly so bad at it simply mounts the wrong file 100% of the time, not even with 50/50 odds of getting it right. lol), skipping the launchbox link check and booting pre-cached games 4x quicker, filling a full 20gig cache and auto deleting the oldest automatically, keeping the one id check to be kept, and deleting the next oldest. Beautiful! I haven't been able to check if its completing RetroAchievement ID scans correctly (although i bet your right that it was this causing the issue) as the emulator im using doesnt have the ability to use them and Retro Achievements are undergoing all sorts of issues and changes on their end that Launchbox is a mess in that regard atm. So yeah thank you very much for what is possible the most useful plug in ive seen for lauchbox (my other favourite bring PCSX2 Configurator Next, which is also an amazing tool) that they should really consider adopting it as standard. Also what is even better is now with this fix even when replacing the standard 7zip file within the 3rd party 7zip folder with a full 64x 7zip install, (where lauchbox unzips work faster) is your cache manager DOES recognize this 7zip .exe / File too!! (Where im pretty sure it didn't seem to before, at all.) So the best of both worlds, (you may even want to mention doing this in your Install instructions as an optional performace boost. ) .Now im super happy. Thank you so much for looking into it.

Further to this i have now made the Archive Cache folder path to an SSD with a 200gig max size, making pre-cached boot times of large files extremely fast... i can only imagine what a NVME or even a Ram Cache could achieve. lol

Thanks for testing and reporting back, and the supportive comments. Happy to hear it's working well now. I'll look at polishing the changes are putting out a new release soon-ish.

@huh123 Sorry for not following up the M3U issue sooner. Is an M3U file being created but its contents are incorrect, or is the M3U not being created at all? Under the additional apps, does each disc have a disc number set?

If you disable Archive Cache Manager (uncheck the Extract ROMs option under the emulator settings for RetroArch > Associated Platforms), does LaunchBox's own M3U playlist creation work with those games?

In answer to your latest questions:

1. Yes, that's the way to add multi-disc games. Archive Cache Manager creates M3U files based on the additional apps list, where one of the additional apps must also be the same as the ROM path for that game (usually disc 1). The M3U creation logic follows LaunchBox's own M3U creation logic as documented here. If the file is an M3U already, it gets treated as a normal ROM file - a future update to parse the M3U file and cache the files it references is something I've wanted to add, but haven't gotten around to yet.
2. Games only appear in the Cache View if there's a game.ini file in the cached folder, which includes some metadata about the extracted / copied game. I'd guess that file doesn't exist, but it's unclear why it wasn't created.

[Baggio]

I've only just come across this and it sounds like exactly what i need for my NAS.

I'm struggling to get it working the way I want though.  From reading your release notes we are able to cache non-archived files and that's what I'm trying to set up.  Basically, I just want it to copy the game from my NAS to my LB SSD on launch, so do I need to set up each emulator in the extraction settings or can I have one setting for them all?

At the minute I can't get it to cache anything, because i can't get my head around it lol

 

Edited September 23, 2023 by Baggio

[Baggio]

I think i may have it actually.  I just need to check "Extract ROM archives before running" in each emulator and set copy in the "All" extraction settings to achieve what I'm after and that's all there is to it. 

 

 

Edited September 23, 2023 by Baggio

[JamesBond@ge]

On 9/9/2023 at 6:46 PM, W4rCh1ld said:

Well ive gotta say that works an absolute treat!!

Fragnator is awesome, mayn! Just sayin'

[sammaxton]

Hey, firstly thanks for this awesome plugin.

I have a tool that I made for adding PC games to Launchbox that are stored as compressed files on a network drive.

In the new Omega version I am working on I am using your Archive Cache Manager to handle the extraction process which creates a cleaner more integrated look for launching the PC games in Launchbox.

The problem I am having is that with some games when they are extracted into the cache folder some of the files are being deleted. This does not happen with all games but there are some that are having this issue. I was hoping you could shed some light onto why the cache manager is deleting some of the files.

For some extra reference to the issue, I had to change the way I was compressing my games to compressing the entire game folder instead of the files within the game folder because when any game was extracted to the cache folder into the game folder created by the cache manager all the root files were being deleted. I was able to fix this by compressing the entire game folder, but as I stated with a few games I am having the issue with all the files being deleted in the game folder.

Examples of the file structures:

The original way was D:\Launchbox\Uncompressed Game Cache\(Name Of Game).7z - E6FE44\(Game Files)

This would result in the game files being deleted.

The workaround I am using D:\Launchbox\Uncompressed Game Cache\(Name Of Game).7z - E6FE44\(Name Of Game) Folder\(Game Files)

This works with most games although there are few that I haven't been able to get to run even though it looks like all the files are there, but more concerning are the ones that the game files are still being deleted.

Any light you might be able to shed on why this is happening would be awesome. I would really like to figure out how to get any PC game working properly with your Cache Manager.

Thanks!

Edited October 23, 2023 by sammaxton

[huh123]

@fraganator Sorry for the late response to your response. I am happy to report I was able to get multi-disc games working. I should have written down exactly what was the issue and how I solved it. I believe my current settings involve creating my own M3U manually and having the games copy rather than extract in the Archive manager settings. I also remember changing the disc settings in Launchbox for the individual titles and discs. Hopefully that helps if anyone else runs into issues!

Unfortunately I did run into another issue recently. If you have games that you wish to softpatch with IPS files of the same name, I cannot get them to work. For example:

I have my base game file, "Game A.game". I have a patch for it, "Game A.ips". If I run the Game A file through Retroarch, it softpatches properly. If I run it through Launchbox, the base game file runs and the soft patch is ignored. I've tried including the patch file in a single .zip or .7z that also includes the base game and have it extract the archive. When I do this and check what the archive manager extracts, it has a folder with both the patch and the base game and the base game does run successfully, but the patch is entirely ignored. Any thoughts/help on this?

Thanks again for this amazing plugin, it's really a fantastic add-on!

[Clutz450]

Is it possible to use zstd zips with this?  RomVault just added support for zstd and I converted a bunch of my ROMs before testing them.  It turns out that RetroArch will not play the zstd ROMs and 7zip doesn't support uncompressed zstd.  MAME already supports zstd and it sounds like this could be the new standard going forward.  What are you thoughts?  Thanks.

[Tayley]

Downloaded and installed sucessfully the plugin, but seems it doesnt work for PS2 games (PCSX2 emulator)

Even with the plugin installed, it attempts to run the .cue file instead of the .bin (see attached images)

Also, havent tested this extensively, but seems none of the games I play appear on the Archive Cache Manager, tried with small and big size games, so I dont really know if the plugin doesnt work for me or im doing something wrong...


SOLVED: Apparently, you have to keep the folder with its original name, or it doesnt work. WinRAR extracted the folder with a sighly different name, and then the plugin itself made the right Folder, but most of the extracted files (that make the plugin work) were still in the initially created folder with the wrong name. Merged both folders into a single one with the correct name, and now everything works!

Edited March 17 by Tayley
Solved myself :D