Retool: a 1G1R tool for Redump and No-Intro

[unexpectedpanda]

Howdy folks,

I've been beavering away at this as a project to learn Python.

Retool converts Redump and No-Intro dats to 1G1R, doing a better job than dat managers with parent/clone dats. It has both GUI and CLI versions.

You'll still need a dat manager to use the files Retool creates, such as CLRMamePro, RomVault, or Romcenter — you just won't need to use their 1G1R modes, as Retool will have already done the work for you.

Why not use CLRMAMEPro or Romcenter's 1G1R mode with a parent/clone dat?

The short version:

Dat managers don't do a great job of picking canonical 1G1R titles, and No-Intro often misses clones in their dats.

The long version:

Historically if you wanted to create a 1G1R set, you'd use a parent/clone dat in combination with a dat manager like CLRMAMEPro or Romcenter. After loading the dat into the dat manager, you'd set your desired regions and region order, and whether or not to filter by languages (assuming the dat has <release> tags properly set up — something which is vanishingly rare). You'd then trust the dat manager to choose the perfect parent title for you from your favorite region, discarding the clones from other regions.

Here's the thing. The parent/clone dat format was created for MAME, to enable one of the ways in which it organizes its ROMs. A "parent" ROM in MAME contains the base or common files for a game, while "clone" ROMs only contain files that are different from the parent. If you load a clone game in MAME, it's smart enough to load the base files from the parent, and any of the modified files it needs from the clone.

Dat manager 1G1R is effectively a hack on top of this system. In 1G1R mode a dat manager uses the parent/clone relationships in a dat to set up a group of related titles. It then selects a single title from that group based on your region and language preferences and ignores the other titles, in an effort to only include the most desired version. While on the surface this seems reasonable, if you look a little closer you start to see the cracks.

The primary issue is that dat managers and parent/clone dats don't have an expansive concept of title priority. For example, what happens when there are two copies of the same title from the same region, but they have different names? Or different version numbers? Or were published by different companies at different times? Which title does the dat manager choose then? The answer — the first one it finds, not the newest or best.

There's another issue, and that concerns languages. To determine a 1G1R title, dat managers give each title a score, combined from your region and language priority. While selected regions are treated as a filter (don't include Europe, and you won't get European titles), languages aren't. Instead, languages are treated as a "bonus" score added to the original region score, meaning you can end up with titles in languages you don't want if a region is high enough priority. Check out LogiqX's pseudo code for an idea of how this works (search for "I do this kind of thing for a living..." to find the relevant bit).

Retool handles proper 1G1R title selection, and a whole lot more. It even identifies the languages of each title by using multiple sources — the implied language spoken in the region the title is from, languages explicitly listed in the title's filename, and languages listed on Redump's website and No-Intro's database, which aren't always included in the filenames.

After you set up the GUI or user-config.yaml to your liking, Retool's output is already 1G1R, meaning you don't need to select 1G1R mode, regions, or languages in your dat manager — just load the dat and go.

Clone lists

While Retool is smart enough to automatically match certain types of parents and clones, there are certain situations that require manual assignment, like when a title has a different name in different regions. To achieve this, Retool keeps clone lists.

At the time of writing, Retool's clone lists are the most thorough that I know of. I manually combed through titles in most dats, and cross referenced them on Wikipedia, Moby Games, Retroplace, GameTDB, VDGB, VGM, YouTube, Amazon.jp, PlayAsia, Sega Retro, PSCX2 Wiki, PlayStation DataCenter, The Cutting Room Floor, and Atari Mania. I checked out the parent/clone dats for No-Intro, and occasionally I went through Redump's site for Japanese, Korean, Russian, and Chinese characters for titles, so I could do translations and find out what they were called in other languages. When all else failed, I did some good old web searching in order to turn up information. At some point I discovered FilterQuest, a similar tool, and added some missing titles from there.

Clone lists are updated independently of the program, and are formatted as JSON files. They are stored in a subfolder called clonelists, which is in the same folder as Retool.

You can update them from the GUI using the File menu, or by running updateclonelists.py.

Features

• Region ordering and language filtering.
• Can exclude the following title types:
  • Add-ons
  • Applications
  • Audio titles
  • Bad dumps
  • BIOS and other chips
  • Bonus discs
  • Coverdiscs
  • Demos and samples
  • Educational titles
  • Manuals
  • Multimedia titles
  • Pirate titles
  • Preproduction titles (alphas, betas, prototypes)
  • Promotional titles
  • Unlicensed titles
  • Video titles
• Custom exclude and include filters.
• Output a list of the final 1G1R title names, and prefix and suffix each title with whatever you like.
• The most thorough clone lists available, which can update from the internet.
• A GUI.
• A legacy parent/clone dat creation mode, and a tool called CloneRel, so you can analyze the relationships Retool creates. This is mostly useful if you intend to update clone lists yourself.

Where can I read more or download Retool?

Over on the Retool Github page, where there are installation instructions for those familiar with Git and Python, and also for Windows users who haven't had much CLI, Git, or Python experience. Read the Wiki for the full set of instructions.

What else is out there?

There are a few different projects that all have slightly different focuses, like Pixelpiper's DCTF, AndreBrait's 1G1R romset generator, and UnluckyForSome's FilterQuest.

 

 

Edited May 21, 2023 by unexpectedpanda

[WraithTDK]

This looks FANTSTIC! Unfortunately, I can't get the GUI to work. I installed Python and followed all the stups, but when I run "retool-gui.py" it looks like a cmd window flutters open momentarily and then closes.

[unexpectedpanda]

On 10/25/2020 at 9:25 AM, WraithTDK said:

This looks FANTSTIC! Unfortunately, I can't get the GUI to work. I installed Python and followed all the stups, but when I run "retool-gui.py" it looks like a cmd window flutters open momentarily and then closes.

This sounds like the dependencies haven't installed correctly. I'll need your assistance in doing some command line work to get things working for you. Hit me up with a private message

Useful to know:

- What OS you're using.
- What version of Python you installed.
- What steps you followed for installation.

Edited November 5, 2020 by unexpectedpanda

[WraithTDK]

6 hours ago, unexpectedpanda said:

This sounds like the dependencies haven't installed correctly. I'll need your assistance in doing some command line work to get things working for you. Hit me up with a private message

Useful to know:

- What OS you're using.
- What version of Python you installed.
- What steps you followed for installation.

I'm grateful for your willingness to help, but I was able to get it working on my laptop (no luck on my desktop, but it was easy enough to transfer the generated DAT files to the desktop). Thanks for making a great program.

[unexpectedpanda]

10 hours ago, WraithTDK said:

I'm grateful for your willingness to help, but I was able to get it working on my laptop (no luck on my desktop, but it was easy enough to transfer the generated DAT files to the desktop). Thanks for making a great program.

Good to know! Sorry my reply took so long, I didn't get a notification on the thread, which means I probably missed a setting somewhere ?

If I were to take a wild stab in the dark, you might have both Python version 2 and 3 installed on the desktop, which could be getting in the way. Or it could be a path issue.

Perhaps I should bundle up an .exe after all, even if it is huge -- it might help avoid some installation issues.

Edited November 6, 2020 by unexpectedpanda

[Solo]

2 hours ago, unexpectedpanda said:

Good to know! Sorry my reply took so long, I didn't get a notification on the thread, which means I probably missed a setting somewhere ?

If I were to take a wild stab in the dark, you might have both Python version 2 and 3 installed on the desktop, which could be getting in the way. Or it could be a path issue.

Perhaps I should bundle up an .exe after all, even if it is huge -- it might help avoid some installation issues.

Hey

 

I have the same issue here Windows 10 1903 and Python 3.8.3

[unexpectedpanda]

8 minutes ago, Solo said:

Hey

 

I have the same issue here Windows 10 1903 and Python 3.8.3

I'll PM you, let's see if we can track down the issue

[unexpectedpanda]

Looks like the command prompt flashing issue is related to having multiple Python installs and Windows not knowing what to do with them. I'll have a binary version up by tomorrow that will take a few seconds to start up, but should bypass this problem altogether.

 

Edit: Okay, there's now a Windows binary available that should make things a bit more friendly for those with this issue. Make sure to update the clone lists from the File menu before using.

Edited November 6, 2020 by unexpectedpanda

[unexpectedpanda]

Retool GUI tutorial

1. Load the Retool GUI, either by running retool-gui.exe or retool-gui.py. Don't close the command prompt window or the program will also close.
   
2. Click File > Check for clone list updates. This is especially important if you've downloaded the Windows binary, as by default clone lists aren't included.
3. If you've got one dat you want to convert to 1G1R, click Choose single dat, then select the file. If you have a whole folder of dats to convert, click Choose folder of dats, then select the folder.
4. Click Choose output folder, then select the folder you want the 1G1R dat to be created in.
5. In the Regions tab, add the regions you want to include in the 1G1R dat to the Filter by these regions field by clicking on the arrow buttons. Tip: You can select multiple regions by holding CTRL when you click, then move them all at once when you click the arrow buttons.
   • The right arrow moves a selected region from the Available regions field to the Filter by these regions field.
   • The left arrow moves a selected region from the Filter by these regions field to the Available regions field.
   • The double right arrow moves the remaining regions in the Available regions field to the end of the list in the Filter by these regions field.
   • The double left arrow moves all regions from the Filter by these regions field into the Available regions field.
   • The up and down arrows change the order of the selection region in the Filter by these regions field.
6. In the Languages tab, move the languages you want to include in the 1G1R dat into the Filter by these languages field by clicking on the arrow buttons. If you leave the Filter by these languages field empty, all languages are included in the 1G1R dat. Tip: Titles with unknown languages are always included.
7. In the Exclusions tab, select the types of titles you would like to exclude. Hover the mouse cursor over each checkbox to learn more about each exclusion type.
8. In the Modes tab, you can select whether or not supersets replace standard editions. This means things like game of the year editions will become the canonical 1G1R title instead of the standard one. If you leave it unchecked, the latest version of all editions are included in the 1G1R dat. You can also choose to export in legacy parent/clone format here, but this is generally only for debugging -- most of the time you'll want to avoid this.
9. Click Go! and watch the command prompt window for progress.
10. Load the 1G1R dat in CLRMAMEPro, RomVault, or Romcenter, and use it as you would an ordinary dat -- ignore the dat manager's 1G1R options.

Got questions? Make sure to check out the Wiki, or feel free to ask here.

Edited November 6, 2020 by unexpectedpanda

[unexpectedpanda]

5 hours ago, Pixelpiper said:

Check https://www.screenscraper.fr/ for a fairly comprehensive clone database.

Looks like you're short the same info I am right now, which is access to Redump's newest platform lists which require a membership - Wii, WiiU, PS3, etc...

 

Cheers

I don't have the language + other metadata for those systems, no. You can still plug the dats in, and build the relevant clone lists, but the language filter won't be as effective as it is on other systems.

[fraganator]

Hi @unexpectedpanda - I just ran across your tool yesterday on github and it's exactly what I needed. I'm currently working on a multi console region LaunchBox setup, and using Retool to come up with USA, Europe, and Japan lists for some No-Intro sets. The results are excellent. The clone lists and extra filter options also really help pare down the results. Really great work

[Ramos]

Hi, nice project.

Is possible add more "Category Types" in Exclusion like (Add-Ons and Bonus Discs)? Very useful in Sony PS Redump Dat

 

Thanks

 

[unexpectedpanda]

On 4/8/2021 at 11:55 AM, fraganator said:

Hi @unexpectedpanda - I just ran across your tool yesterday on github and it's exactly what I needed. I'm currently working on a multi console region LaunchBox setup, and using Retool to come up with USA, Europe, and Japan lists for some No-Intro sets. The results are excellent. The clone lists and extra filter options also really help pare down the results. Really great work

Glad it's working well for you

18 hours ago, Ramos said:

Hi, nice project.

Is possible add more "Category Types" in Exclusion like (Add-Ons and Bonus Discs)? Very useful in Sony PS Redump Dat

 

Thanks

 

Yes, it's possible. I'll look into it.

[unexpectedpanda]

On 4/9/2021 at 12:04 AM, Ramos said:

Hi, nice project.

Is possible add more "Category Types" in Exclusion like (Add-Ons and Bonus Discs)? Very useful in Sony PS Redump Dat

 

Thanks

 

These are now in as of v0.89.

Edited April 14, 2021 by unexpectedpanda

[Ramos]

13 minutes ago, unexpectedpanda said:

These are now in as of v0,89.

I tested it and it worked perfectly.
Excellent work.

Thank you.

[JoseJL]

This is an amazing tool, great job.

Have you considered adding some kind of support for MAME software list XMLs?

I'm working on some Python script to match them as best as possible
 

Edited April 24, 2021 by JoseJL

[unexpectedpanda]

2 hours ago, JoseJL said:

This is an amazing tool, great job.

Have you considered adding some kind of support for MAME software list XMLs?

I'm working on some Python script to match them as best as possible
 

Cheers! Support is limited to Redump and No-Intro -- that's more than enough to occupy my time :)

[zetec-s-joe]

Great tool

[Hazuki]

Hi,

First off, fantastic tool. Have just been trying it out and I'll be switching to this for my previous 1G1R tool. Thanks you for this

Just one little question. I can't find any option for the "Compilations with no unique titles". Am I missing something or is this something I'll add as a Custom golbal/system filter?

[unexpectedpanda]

26 minutes ago, Hazuki said:

Hi,

First off, fantastic tool. Have just been trying it out and I'll be switching to this for my previous 1G1R tool. Thanks you for this

Just one little question. I can't find any option for the "Compilations with no unique titles". Am I missing something or is this something I'll add as a Custom golbal/system filter?

Cheers

A lot has changed since the original post! As of v0.89 the compilations and supersets options were removed, and they're now on by default. I'll quote myself from the changelog at the time:

Quote

Things started getting confusing with compilations and supersets when it came to choosing a 1G1R title. This was always an incredibly grey and fuzzy area, was a pain to maintain, and it became apparent over time that things needed to change for the sake of clarity. The upshot of all this? You get a better 1G1R selection without having to select the right options.

Things that have changed:

• Mega-CD 32x now gets hoisted above Sega CD 32x if you have Europe above USA in your region order.
• A removes key is now available to use in clone lists to remove specific titles from a dat. It works with full, tag free, or short names. This is mostly useful to take out titles that don't quite match anything, but should still be removed — for example, compilations whose titles are covered by other compilations, or individual titles in the same region.
• The supersets option has been removed. It was nice to have in theory, but in practice, game of the year editions, special editions, and so on are mostly just the latest versions of games with bonus content and/or DLC included. There are incredibly rare exceptions — such as Ninja Gaiden vs Ninja Gaiden Black, where dramatic rebalancing was done in addition to the extra content, along with new enemies and weapons — but for the most part, this isn't the case. If you care enough about an older version of a game, you can always add it to a custom system filter. Supersets are now often assigned a 1 or 0 priority, depending on the situation.
• The exclude compilations option has been removed, and compilations are now usually treated in one of two ways:
  • As a remove; that is, the compilation is removed entirely from the dat as it's covered by other titles or compilations.
  • As the 1G1R title for a set, depending on the region.

There's a bunch of new features in there too, so I'll make some time to update the original post in the future.

Edited June 1, 2021 by unexpectedpanda