Unofficial RetroAchievements 1G1R DATs and CUEs/GDIs. 03 January 2025 Update 2

Cool, no worries about rclone. I may look into it though at later stage. But clearly not important.

So PS2 roms would need to be unzipped to be playable from launchbox/emulationstation. Same for some others and also some may require bios files or patches? Let me look into this how to find my routine. In this case, is always need to keep the rom three times (original, processed with cmp, unzipped).

2 minutes ago, steve1977 said:

Cool, no worries about rclone. I may look into it though at later stage. But clearly not important.

So PS2 roms would need to be unzipped to be playable from launchbox/emulationstation. Same for some others and also some may require bios files or patches? Let me look into this how to find my routine. In this case, is always need to keep the rom three times (original, processed with cmp, unzipped).

With disc-based platforms (PS1, PS2, PSP, GC, etc), I usually keep 2 copies, the zipped copy to scan with cmp and the unzipped copy to import into LB and run with the respective emulator.

47 minutes ago, Ultra God Azgorath said:

What do you mean by rim?

Typo. Rom.

40 minutes ago, Ultra God Azgorath said:

With disc-based platforms (PS1, PS2, PSP, GC, etc), I usually keep 2 copies, the zipped copy to scan with cmp and the unzipped copy to import into LB and run with the respective emulator.

Just to understand your workflow, how do you make sure that you remember that you already scanned the zip before? CMP will mark it as missing after you deleted the CMP sorted file?

10 minutes ago, steve1977 said:

Just to understand your workflow, how do you make sure that you remember that you already scanned the zip before? CMP will mark it as missing after you deleted the CMP sorted file?

So I have 2 rom folders on my pc. 1 folder, which is my main rom folder, contains cartridge-based platforms with zipped roms that are never unzipped and disc-based platforms with unzipped roms. I use cmp to scan only the cartridge-based platforms with the cartridge-based DATs in this folder. I import all the roms in this folder into LB. The 2nd folder contains disc-based platforms with zipped roms and I use cmp to only scan this folder with the disc-based DATs. CMP doesn't check your collection in real-time, only when you manually scan. If you look at the attached picture, you'll see that the platforms with the red circle are missing some roms while the platforms with green ticks means all roms were scanned and found. By keeping the directories saved and the roms saved in their respective directories, when I manually scan the folders, cmp will tell me if any roms are missing or not.

Thanks! In this case, you delete the unmatched ones? And redownload once they’re added to RA?

I still don’t fully understand, but let me play around it a bit over the weekend. You put a lot of thinking into your workflow, so a lot to learn from. Thank you!

2 hours ago, steve1977 said:

Thanks! In this case, you delete the unmatched ones? And redownload once they’re added to RA?

I still don’t fully understand, but let me play around it a bit over the weekend. You put a lot of thinking into your workflow, so a lot to learn from. Thank you!

cmp by default moves unmatched roms into a backup folder or you can tell it to delete them permanently instead. When it comes to adding and deleting games from the DATs, I add new games that are added on the RA homepage to the DATs but when they're removed and/or re-added after being removed for maintenance, I dont currently have a way to tell when they're removed and/or re-added from maintenance.

I've been playing a bit around with both CMP and RomVault. The latter, I found a lot easier to use.

I am having the same issue though with both tools. It shows all roms, but doesn't group them. I.e., I see 3 roms for 1942 (NES), but it doesn't merge them into one game. For example, your screenshot shows 1,047 NES roms. CMP shows 1,789 sets and 1,937 roms. RomVault shows 1,884 roms. All using your file.

I also did some checks. For example, "10-Yard Fight (Japan).zip" shows up in your file, but it is not listed on the RA web-site. Would this be an example of a file that has been removed from RA, which you mentioned that you cannot spot?

3 hours ago, steve1977 said:

I also did some checks. For example, "10-Yard Fight (Japan).zip" shows up in your file, but it is not listed on the RA web-site. Would this be an example of a file that has been removed from RA, which you mentioned that you cannot spot?

Yes that's correct. I've removed it from the DAT now, thanks but there's probably a lot more like this in a lot of the other dats as well (I hope not though).

3 hours ago, steve1977 said:

I am having the same issue though with both tools. It shows all roms, but doesn't group them. I.e., I see 3 roms for 1942 (NES), but it doesn't merge them into one game. For example, your screenshot shows 1,047 NES roms. CMP shows 1,789 sets and 1,937 roms. RomVault shows 1,884 roms. All using your file.

cmp shows 1047 for me because I have 1G1R enabled with english regions prioritised. Romvault doesnt have 1G1R mode/feature so it looks for every single game that's in the DAT, in your rom folder. As for the 1789 sets and 1937 roms, I have no idea whats going wrong there because I've checked my DAT and there's 1883 entries (Sets) in the DAT and each set has 1 rom in it so you should be getting 1883 sets and 1883 roms, not 1789 and 1937. Each rom needs to be it's own entry, not grouped with others.

Edited January 4 by Ultra God Azgorath

Thanks for sharing. I played around a bit more with both. I will still go ahead with RomVault. It even has a feature that allows to unzip the disc-based files and still count it against the DAT. So, I don't need to keep duplicates. It has some form of "merge mode" that at least things are grouped and the list only includes 1047. So, it still shows xxx/1884 rather than xxx/1047. Think that's fine though and I may miss a feature that allows to display 1047. Have you tried RomVault yourself? Its seems much easier to use and will help you remove the zip duplicates.

Do you keep the roms that have not been matched? If so, this is taking a lot of space. But if not, this probably requires you to re-download files once they are added. How often does RA adds new sets, so much new download-work is required every month?

If I keep two region files in one folder (xx-eu.iso and xx-us.iso), do you know whether LB/ES will give me the option, which version to play? If so, will it still count both against the same RA record?

When it comes to the removed dats, what are the consequences? Game still plays, but there is no RA? What's the reason why RA removes them? Likely many of them as I stumbled over the file above by just checking 3 games.

3 hours ago, steve1977 said:

Have you tried RomVault yourself? Its seems much easier to use and will help you remove the zip duplicates.

I have it downloaded and have tried it but cmp fits my needs better.

3 hours ago, steve1977 said:

Do you keep the roms that have not been matched? If so, this is taking a lot of space. But if not, this probably requires you to re-download files once they are added. How often does RA adds new sets, so much new download-work is required every month?

No I delete the unmatched roms so yeah I have to redownload them if and when they're re-added. New games are added daily so I put new games in the DATs every day.

3 hours ago, steve1977 said:

If I keep two region files in one folder (xx-eu.iso and xx-us.iso), do you know whether LB/ES will give me the option, which version to play? If so, will it still count both against the same RA record?

Yes, LB combines both regions into 1 game and you right click the game and click "Play Version" and you select the version you want to play. As long as both regions are linked to the same achievement set, they'll both work with achievements.

3 hours ago, steve1977 said:

When it comes to the removed dats, what are the consequences? Game still plays, but there is no RA? What's the reason why RA removes them? Likely many of them as I stumbled over the file above by just checking 3 games.

There's a few reasons why games would be removed from RA.

1. There is 1 or more achievements that are broken and need to be repaired by the dev so the dev temporarily removes (demotes) the set to fix it and then promotes it back once he's fixed it.
2. The RA team decided that an achievement set for a game is not worth having on the site for whatever reason.
3. The ra team and/or dev decided to only have 1 region rom linked to a set. So instead of having USA and EUROPE both linked, they decide to keep only EUROPE linked since EUROPE has more languages and USA doesn't have anything that EUROPE doesn't already have.

There may be other reasons but these are the most common reasons I've seen.

Got it, very helpful!

RomVault is now running for almost 24 hours, so should soon see where things stand.

If I have two regions and just select the game, what region does it play (i.e., not right-clicking).

Still need to decide on LB vs ES-DE. I believe I had purchased LB some decade ago,but then moved to ES-DE. Forgot the reason. Have you tried both and a preference?

If you delete the unmatched, you also have to download the newly added ones every day? That seems quite an effort? Besides PS2/GC (rom size issue), wouldn't it be more efficient to keep the unmatched roms in the backup folder to add them automatically once RA adds support?

6 minutes ago, steve1977 said:

If I have two regions and just select the game, what region does it play (i.e., not right-clicking).

I'm not sure which region LB plays 1st. I think its alphabetical so E before U but I could be wrong.

6 minutes ago, steve1977 said:

Still need to decide on LB vs ES-DE. I believe I had purchased LB some decade ago,but then moved to ES-DE. Forgot the reason. Have you tried both and a preference?

I've tried ES-DE. Awesome frontend. Very easy and fast but no retroachievement features yet. Once ES-DE gets RA features, I'll be switching over to it. I'm only still using LB because of the RA features.

6 minutes ago, steve1977 said:

If you delete the unmatched, you also have to download the newly added ones every day? That seems quite an effort? Besides PS2/GC (rom size issue), wouldn't it be more efficient to keep the unmatched roms in the backup folder to add them automatically once RA adds support?

Yes but I don't mind doing it this way. I only want RA-compatible games so I'll download them as they're added.

Thanks, I thought I cracked it with the "merger" function. Unfortunatelty, it does not work for all platforms. Any thoughts?

NES
1942 (Japan, USA) (En).zip including 1942 (Japan, USA) (En).nes, 1942 (Japan, USA) (Es) (Emu4ever) (v0.98).nes, and 1942 (Japan, USA) (Fr) (ks151) (v1.0).nes

Apple II
Separate zip files for Black Cauldron, The (1985)(Sierra On-Line)(IIE)(US)(Disk 1 of 3 Side A)[cr 4am - san inc][128K].zip, Black Cauldron, The (1985)(Sierra On-Line)(IIE)(US)(Disk 1 of 3 Side B)[128K].zip, Black Cauldron, The (1985)(Sierra On-Line)(IIE)(US)(Disk 2 of 3 Side A)[128K].zip,etc.

2 hours ago, steve1977 said:

Thanks, I thought I cracked it with the "merger" function. Unfortunatelty, it does not work for all platforms. Any thoughts?

I never use the merger option, only the scanner option.

So, all your files with different regions remain separate? Do they show up as multiple games or does LB recognize them and "merges" them for you?

NES show 1,884 sets in RomVault, which seems the same for you in CMP?

Apple II shows 181 sets for me. Assume this is less for you in CMP?

Edited Monday at 12:52 AM by steve1977

7 hours ago, steve1977 said:

So, all your files with different regions remain separate? Do they show up as multiple games or does LB recognize them and "merges" them for you?

NES show 1,884 sets in RomVault, which seems the same for you in CMP?

Apple II shows 181 sets for me. Assume this is less for you in CMP?

During import, LB will give you an option to tick if you want to combine games with the same or similar name or keep them separate. Yes cmp keeps all files separate if you have normal set mode selected, otherwise if you have 1g1r mode selected, cmp will keep the roms with the regions you selected and prioritized and delete/backup the others.

Edited Monday at 06:49 AM by Ultra God Azgorath

Just now, Ultra God Azgorath said:

Apple II shows 181 sets for me. Assume this is less for you in CMP?

It's 128 for me because of 1g1r.

The odd part with romvault is that it follows 1g1r for some platforms, while not for others. For example, NES and SNES follow the logic of 1g1r. I can keep the region files separately or even include them into the same zip file (whatever i prefer). However, apple ii for whatever reason doesn’t follow the same logic and the region files stay separate and count separate.

Amy idea how the DAT file for nes/snes is different from the one for apple ii?

The only difference between apple (also amstrad CPC) DATs and the other platforms DATs is the regions themselves. Apple and CPC don't have normal regions such as USA, EUROPE, JAPAN, etc and because of that, I named their regions after the group they were dumped from (4AM, CLEANCPCDB, TOSEC, GOODSET, etc)

Ah… This explains… So, nes is picked up as regions by romvault based on how they’re named and not based on 1g1r. And that’s why apple is not being picked up as the naming convention doesn’t allow to be identified as region. Though clearly some of them are regions (eu and us in my example above), the naming convention is different.

Technically, you didn’t name them? It’s the file naming from RA?

Probably same reason why the discs aren’t grouped. Amstrad disks are not clean (disk a1, etc), so romvault doesn’t pick them up as being part of the same set.

Im surprised that launchbox eventually is able to identify them as one set when importing them.

8 minutes ago, steve1977 said:

Technically, you didn’t name them? It’s the file naming from RA?

Yes and no. Some are from an "unofficial" apple/cpc DAT but others are named from RA but I couldn't find/figure out which roms have which regions so I had to improvise.

Edited Monday at 11:54 AM by Ultra God Azgorath

Elektor tv games computer has the same problem because there is no official dat for that platform for me to build on so the regions are the roms' extensions (PGM and TVC).

Wouldn’t it be best to take the official file name? The name on RA comes from Redump, No-Intro, etc. And it’s what’s on the RA web-site. Is it about the effort to copy&pasts from RA web-site? Or some other reason that’s driving the name choice?

With the No-intro, fbneo and redump platforms, I copy and paste the entries straight from the official dats to my dats. Amstrad CPC, Apple II, PC 8000/8800 and Elektor are the exceptions due to either not having an official DAT or whatever DATs I could find for them weren't quite as "clean" as the no-intro and redump dats.