Scan for RetroAchievements Issue

[moudrost]

I'm on 13.9 and having RA back is most welcome. I started to scan each platform so LaunchBox will match to the achievement profile. Works great for all non-CD based games. None of the CD based systems have matched, however the ones I know are valid are identified correctly in RetroArch. Is there something I'm doing wrong here? I have used TorrentZip on all my images and match all recent DATs. So far the issue only seems to be with optical media. Any help would be appreciated.

[retroNUC]

Had a quick check of the RAHasher code, which is the tool that LB uses to generate hashes. Confirmed it doesn't support disc-based files within ZIP archives, and that's for a good reason - For each game, it would have to spend a while to fully extract the entire ZIP file somewhere temporarily just so it can mount the CD and read the specific files it needs to within it. You can't do that while it's still packed in an archive file.

You really shouldn't be zipping disc-based games, to be honest. It might work, but you're probably getting a really slow launch every time you try to run a game? Either LaunchBox or the emulator will be doing a full unzip/extract.

Use other compression formats if you want to save space, pretty much everything supports CHD these days. Same kind of algorithm as a zip file, but only on the inner data and not the file headers needed for that random file access.

https://retrogamecorps.com/2023/02/06/the-ultimate-rom-file-compression-guide/

[atlantisworld]

I have updated 13.9, indicated retroachievements credentials and after doing a full scan, I can only see that it appears in the Arcade section, and Nintendo 64, but on the rest of the systems it does not appear, for example Shinobi (USA, Europe).gg (Sega Game Gear), Genesis, Nes... however, when running the game, Retroachievements does appear in Retroarch. I would be interested in seeing it appear on all compatible systems in both Launchbox and BigBox. What am I doing wrong?

[moudrost]

23 hours ago, retroNUC said:

Had a quick check of the RAHasher code, which is the tool that LB uses to generate hashes. Confirmed it doesn't support disc-based files within ZIP archives, and that's for a good reason - For each game, it would have to spend a while to fully extract the entire ZIP file somewhere temporarily just so it can mount the CD and read the specific files it needs to within it. You can't do that while it's still packed in an archive file.

You really shouldn't be zipping disc-based games, to be honest. It might work, but you're probably getting a really slow launch every time you try to run a game? Either LaunchBox or the emulator will be doing a full unzip/extract.

Use other compression formats if you want to save space, pretty much everything supports CHD these days. Same kind of algorithm as a zip file, but only on the inner data and not the file headers needed for that random file access.

https://retrogamecorps.com/2023/02/06/the-ultimate-rom-file-compression-guide/

To each their own I guess on zip vs CHD. I'm not storing them in that format to save space. Converting to CHD is a non-starter. It would be impossible to manage my library in CHD. Even if Redump started to release DATs compatible with CHDs, I could only imagine how long it would take to scan/fix a full set. Part of the advantage of using ZIP over 7z is so it doesn't need to decompress the entire file to calculate the hash. Is this a bug with the tool used to calculate the hashes?

I use this wonderful plugin which pretty much alleviates waiting on decompressing for my most used games.

I set aside 500GB as a temporary space for the decompressed archives. As I run out of space in the cache, it automatically deletes the oldest used files.

 

[C-Beats]

On 12/16/2023 at 10:12 PM, retroNUC said:

Had a quick check of the RAHasher code, which is the tool that LB uses to generate hashes. Confirmed it doesn't support disc-based files within ZIP archives, and that's for a good reason - For each game, it would have to spend a while to fully extract the entire ZIP file somewhere temporarily just so it can mount the CD and read the specific files it needs to within it. You can't do that while it's still packed in an archive file.

You really shouldn't be zipping disc-based games, to be honest. It might work, but you're probably getting a really slow launch every time you try to run a game? Either LaunchBox or the emulator will be doing a full unzip/extract.

Use other compression formats if you want to save space, pretty much everything supports CHD these days. Same kind of algorithm as a zip file, but only on the inner data and not the file headers needed for that random file access.

https://retrogamecorps.com/2023/02/06/the-ultimate-rom-file-compression-guide/

I'll have to review the code. I was fairly certain we unzipped games before running them through the hasher, but it's been a minute since I have looked into the nitty gritty of that logic. @retroNUC weren't you the one that a while back had to compile your own version of RAHasher to work with CHD until they got it pulled in? If so would it be possible for you to check that those changes are still in? We are getting some reports of CHDs hashing differently via that app than they are via the emulator.

[atlantisworld]

I have found in the xml of each platform that the line: <RetroAchievementsHash>COULDNTFILEHASH</RetroAchievementsHash> appears in each game, which prevents the Retroachievements badge from appearing and which is not solved by scanning the games. I have been able to verify that deleting all the lines in which COULDNTFILEHASH appears in each game and scanning the games solves the problem. Is there a way to fix this automatically from the Retroachievements scanner?

[C-Beats]

28 minutes ago, atlantisworld said:

I have found in the xml of each platform that the line: <RetroAchievementsHash>COULDNTFILEHASH</RetroAchievementsHash> appears in each game, which prevents the Retroachievements badge from appearing and which is not solved by scanning the games. I have been able to verify that deleting all the lines in which COULDNTFILEHASH appears in each game and scanning the games solves the problem. Is there a way to fix this automatically from the Retroachievements scanner?

This is only cleared by the app when we update the hasher application we package with. Otherwise the result wouldn't change. Only other time this is cleared is when you change the app path of a game.

[retroNUC]

On 12/18/2023 at 3:43 AM, moudrost said:

Part of the advantage of using ZIP over 7z is so it doesn't need to decompress the entire file to calculate the hash. Is this a bug with the tool used to calculate the hashes?

No, not a bug. It all depends exactly how a game is hashed - https://docs.retroachievements.org/Game-Identification/

ZIP is fine for ROM-based games, because it's really easy to access and hash a single whole file within a ZIP archive. But for disc images, the hasher needs to be able to mount the image and use the disc header data in order to then access and hash specific files within the image. It's impossible to access a file within an image when the entire image is compressed, so no choice but to spend the 30-60s extracting the full thing first.

 

On 12/18/2023 at 3:43 AM, moudrost said:

Converting to CHD is a non-starter. It would be impossible to manage my library in CHD. Even if Redump started to release DATs compatible with CHDs, I could only imagine how long it would take to scan/fix a full set.

Then the easy solution is to switch clrmamepro to a name-only scan instead of name+hash. Disc-based collections are pretty static these days, it's not like MAME where parent/clone files within a zip constantly change.

Or if you still want to hash things, just use one of the tools that does support CHD, like this one. CHD files store the original's MD5 hash in their header, so it's possible for files to expand and verify themselves for data integrity (and then compare that same hash against redump dats, like the tool I linked does).

 

On 12/18/2023 at 2:45 PM, C-Beats said:

If so would it be possible for you to check that those changes are still in? We are getting some reports of CHDs hashing differently via that app than they are via the emulator.

Far as I can tell, nothing has changed in RAHasher code since the last time those CHD fixes went in 8 months ago, and I imagine nothing would change unless you updated the included binary. Only way I can imagine it going wrong when called by LB was if the wrong Console ID was being passed in?

[C-Beats]

33 minutes ago, retroNUC said:

Far as I can tell, nothing has changed in RAHasher code since the last time those CHD fixes went in 8 months ago, and I imagine nothing would change unless you updated the included binary. Only way I can imagine it going wrong when called by LB was if the wrong Console ID was being passed in?

None of that changed for us, but gives me something to vet against. Thanks for the additional information, much appreciated.

[moudrost]

1 hour ago, retroNUC said:

Then the easy solution is to switch clrmamepro to a name-only scan instead of name+hash. Disc-based collections are pretty static these days, it's not like MAME where parent/clone files within a zip constantly change.

Or if you still want to hash things, just use one of the tools that does support CHD, like this one. CHD files store the original's MD5 hash in their header, so it's possible for files to expand and verify themselves for data integrity (and then compare that same hash against redump dats, like the tool I linked does).

I appreciate you taking the time to reply. I had been using Clrmamepro for some 20 years, but moved over to RomVault. Once library size reaches a certain threshold, Clrmamepro is painful to use. If you pay for RomVault, which I highly recommend, DAT management becomes a real time saver. I have tried a few times to test how long it would take to migrate to CHD and it just isn't worth the investment. Collecting and maintaining games has its own rewards as does playing the actual games and CHDs make maintaining a bit too cumbersome.

With all that being said, is there a reason LaunchBox isn't decompressing the zip archives to generate the hash?

Edited December 20, 2023 by moudrost

[C-Beats]

13 hours ago, moudrost said:

With all that being said, is there a reason LaunchBox isn't decompressing the zip archives to generate the hash?

Fairly certain that we are. We extract them to our temp directory (\\LaunchBox\Metadata\Temp) and then run the hasher against it. I'll review the code and let you know for certain.

[moudrost]

15 minutes ago, C-Beats said:

Fairly certain that we are. We extract them to our temp directory (\\LaunchBox\Metadata\Temp) and then run the hasher against it. I'll review the code and let you know for certain.

When I scanned them the first time, it moved too fast for it to be decompressing any archive. Now for any platform that I have already scanned, it completes in 2 seconds. Looking at the XML for each ISO platform, every record looks the same.

<RetroAchievementsHash>COULDNTFILEHASH</RetroAchievementsHash>

Thanks for looking into it.

 

[C-Beats]

I looked into the code and have discovered we only do the unzip process for .7z files since the hasher can never consume those directly. We DON'T for zip files. I'll look into the code to see how best to address this since we'd need to ensure we only did that for disc based games.