moudrost Posted December 15, 2023 Share Posted December 15, 2023 I'm on 13.9 and having RA back is most welcome. I started to scan each platform so LaunchBox will match to the achievement profile. Works great for all non-CD based games. None of the CD based systems have matched, however the ones I know are valid are identified correctly in RetroArch. Is there something I'm doing wrong here? I have used TorrentZip on all my images and match all recent DATs. So far the issue only seems to be with optical media. Any help would be appreciated. Quote Link to comment Share on other sites More sharing options...
retroNUC Posted December 17, 2023 Share Posted December 17, 2023 Had a quick check of the RAHasher code, which is the tool that LB uses to generate hashes. Confirmed it doesn't support disc-based files within ZIP archives, and that's for a good reason - For each game, it would have to spend a while to fully extract the entire ZIP file somewhere temporarily just so it can mount the CD and read the specific files it needs to within it. You can't do that while it's still packed in an archive file. You really shouldn't be zipping disc-based games, to be honest. It might work, but you're probably getting a really slow launch every time you try to run a game? Either LaunchBox or the emulator will be doing a full unzip/extract. Use other compression formats if you want to save space, pretty much everything supports CHD these days. Same kind of algorithm as a zip file, but only on the inner data and not the file headers needed for that random file access. https://retrogamecorps.com/2023/02/06/the-ultimate-rom-file-compression-guide/ Quote Link to comment Share on other sites More sharing options...
atlantisworld Posted December 17, 2023 Share Posted December 17, 2023 I have updated 13.9, indicated retroachievements credentials and after doing a full scan, I can only see that it appears in the Arcade section, and Nintendo 64, but on the rest of the systems it does not appear, for example Shinobi (USA, Europe).gg (Sega Game Gear), Genesis, Nes... however, when running the game, Retroachievements does appear in Retroarch. I would be interested in seeing it appear on all compatible systems in both Launchbox and BigBox. What am I doing wrong? Quote Link to comment Share on other sites More sharing options...
moudrost Posted December 18, 2023 Author Share Posted December 18, 2023 23 hours ago, retroNUC said: Had a quick check of the RAHasher code, which is the tool that LB uses to generate hashes. Confirmed it doesn't support disc-based files within ZIP archives, and that's for a good reason - For each game, it would have to spend a while to fully extract the entire ZIP file somewhere temporarily just so it can mount the CD and read the specific files it needs to within it. You can't do that while it's still packed in an archive file. You really shouldn't be zipping disc-based games, to be honest. It might work, but you're probably getting a really slow launch every time you try to run a game? Either LaunchBox or the emulator will be doing a full unzip/extract. Use other compression formats if you want to save space, pretty much everything supports CHD these days. Same kind of algorithm as a zip file, but only on the inner data and not the file headers needed for that random file access. https://retrogamecorps.com/2023/02/06/the-ultimate-rom-file-compression-guide/ To each their own I guess on zip vs CHD. I'm not storing them in that format to save space. Converting to CHD is a non-starter. It would be impossible to manage my library in CHD. Even if Redump started to release DATs compatible with CHDs, I could only imagine how long it would take to scan/fix a full set. Part of the advantage of using ZIP over 7z is so it doesn't need to decompress the entire file to calculate the hash. Is this a bug with the tool used to calculate the hashes? I use this wonderful plugin which pretty much alleviates waiting on decompressing for my most used games. I set aside 500GB as a temporary space for the decompressed archives. As I run out of space in the cache, it automatically deletes the oldest used files. Quote Link to comment Share on other sites More sharing options...
C-Beats Posted December 18, 2023 Share Posted December 18, 2023 On 12/16/2023 at 10:12 PM, retroNUC said: Had a quick check of the RAHasher code, which is the tool that LB uses to generate hashes. Confirmed it doesn't support disc-based files within ZIP archives, and that's for a good reason - For each game, it would have to spend a while to fully extract the entire ZIP file somewhere temporarily just so it can mount the CD and read the specific files it needs to within it. You can't do that while it's still packed in an archive file. You really shouldn't be zipping disc-based games, to be honest. It might work, but you're probably getting a really slow launch every time you try to run a game? Either LaunchBox or the emulator will be doing a full unzip/extract. Use other compression formats if you want to save space, pretty much everything supports CHD these days. Same kind of algorithm as a zip file, but only on the inner data and not the file headers needed for that random file access. https://retrogamecorps.com/2023/02/06/the-ultimate-rom-file-compression-guide/ I'll have to review the code. I was fairly certain we unzipped games before running them through the hasher, but it's been a minute since I have looked into the nitty gritty of that logic. @retroNUC weren't you the one that a while back had to compile your own version of RAHasher to work with CHD until they got it pulled in? If so would it be possible for you to check that those changes are still in? We are getting some reports of CHDs hashing differently via that app than they are via the emulator. Quote Link to comment Share on other sites More sharing options...
atlantisworld Posted December 19, 2023 Share Posted December 19, 2023 I have found in the xml of each platform that the line: <RetroAchievementsHash>COULDNTFILEHASH</RetroAchievementsHash> appears in each game, which prevents the Retroachievements badge from appearing and which is not solved by scanning the games. I have been able to verify that deleting all the lines in which COULDNTFILEHASH appears in each game and scanning the games solves the problem. Is there a way to fix this automatically from the Retroachievements scanner? Quote Link to comment Share on other sites More sharing options...
C-Beats Posted December 19, 2023 Share Posted December 19, 2023 28 minutes ago, atlantisworld said: I have found in the xml of each platform that the line: <RetroAchievementsHash>COULDNTFILEHASH</RetroAchievementsHash> appears in each game, which prevents the Retroachievements badge from appearing and which is not solved by scanning the games. I have been able to verify that deleting all the lines in which COULDNTFILEHASH appears in each game and scanning the games solves the problem. Is there a way to fix this automatically from the Retroachievements scanner? This is only cleared by the app when we update the hasher application we package with. Otherwise the result wouldn't change. Only other time this is cleared is when you change the app path of a game. Quote Link to comment Share on other sites More sharing options...
retroNUC Posted December 19, 2023 Share Posted December 19, 2023 On 12/18/2023 at 3:43 AM, moudrost said: Part of the advantage of using ZIP over 7z is so it doesn't need to decompress the entire file to calculate the hash. Is this a bug with the tool used to calculate the hashes? No, not a bug. It all depends exactly how a game is hashed - https://docs.retroachievements.org/Game-Identification/ ZIP is fine for ROM-based games, because it's really easy to access and hash a single whole file within a ZIP archive. But for disc images, the hasher needs to be able to mount the image and use the disc header data in order to then access and hash specific files within the image. It's impossible to access a file within an image when the entire image is compressed, so no choice but to spend the 30-60s extracting the full thing first. On 12/18/2023 at 3:43 AM, moudrost said: Converting to CHD is a non-starter. It would be impossible to manage my library in CHD. Even if Redump started to release DATs compatible with CHDs, I could only imagine how long it would take to scan/fix a full set. Then the easy solution is to switch clrmamepro to a name-only scan instead of name+hash. Disc-based collections are pretty static these days, it's not like MAME where parent/clone files within a zip constantly change. Or if you still want to hash things, just use one of the tools that does support CHD, like this one. CHD files store the original's MD5 hash in their header, so it's possible for files to expand and verify themselves for data integrity (and then compare that same hash against redump dats, like the tool I linked does). On 12/18/2023 at 2:45 PM, C-Beats said: If so would it be possible for you to check that those changes are still in? We are getting some reports of CHDs hashing differently via that app than they are via the emulator. Far as I can tell, nothing has changed in RAHasher code since the last time those CHD fixes went in 8 months ago, and I imagine nothing would change unless you updated the included binary. Only way I can imagine it going wrong when called by LB was if the wrong Console ID was being passed in? Quote Link to comment Share on other sites More sharing options...
C-Beats Posted December 20, 2023 Share Posted December 20, 2023 33 minutes ago, retroNUC said: Far as I can tell, nothing has changed in RAHasher code since the last time those CHD fixes went in 8 months ago, and I imagine nothing would change unless you updated the included binary. Only way I can imagine it going wrong when called by LB was if the wrong Console ID was being passed in? None of that changed for us, but gives me something to vet against. Thanks for the additional information, much appreciated. Quote Link to comment Share on other sites More sharing options...
moudrost Posted December 20, 2023 Author Share Posted December 20, 2023 (edited) 1 hour ago, retroNUC said: Then the easy solution is to switch clrmamepro to a name-only scan instead of name+hash. Disc-based collections are pretty static these days, it's not like MAME where parent/clone files within a zip constantly change. Or if you still want to hash things, just use one of the tools that does support CHD, like this one. CHD files store the original's MD5 hash in their header, so it's possible for files to expand and verify themselves for data integrity (and then compare that same hash against redump dats, like the tool I linked does). I appreciate you taking the time to reply. I had been using Clrmamepro for some 20 years, but moved over to RomVault. Once library size reaches a certain threshold, Clrmamepro is painful to use. If you pay for RomVault, which I highly recommend, DAT management becomes a real time saver. I have tried a few times to test how long it would take to migrate to CHD and it just isn't worth the investment. Collecting and maintaining games has its own rewards as does playing the actual games and CHDs make maintaining a bit too cumbersome. With all that being said, is there a reason LaunchBox isn't decompressing the zip archives to generate the hash? Edited December 20, 2023 by moudrost Quote Link to comment Share on other sites More sharing options...
C-Beats Posted December 20, 2023 Share Posted December 20, 2023 13 hours ago, moudrost said: With all that being said, is there a reason LaunchBox isn't decompressing the zip archives to generate the hash? Fairly certain that we are. We extract them to our temp directory (\\LaunchBox\Metadata\Temp) and then run the hasher against it. I'll review the code and let you know for certain. Quote Link to comment Share on other sites More sharing options...
moudrost Posted December 20, 2023 Author Share Posted December 20, 2023 15 minutes ago, C-Beats said: Fairly certain that we are. We extract them to our temp directory (\\LaunchBox\Metadata\Temp) and then run the hasher against it. I'll review the code and let you know for certain. When I scanned them the first time, it moved too fast for it to be decompressing any archive. Now for any platform that I have already scanned, it completes in 2 seconds. Looking at the XML for each ISO platform, every record looks the same. <RetroAchievementsHash>COULDNTFILEHASH</RetroAchievementsHash> Thanks for looking into it. Quote Link to comment Share on other sites More sharing options...
C-Beats Posted December 20, 2023 Share Posted December 20, 2023 I looked into the code and have discovered we only do the unzip process for .7z files since the hasher can never consume those directly. We DON'T for zip files. I'll look into the code to see how best to address this since we'd need to ensure we only did that for disc based games. 1 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.