Jump to content
LaunchBox Community Forums
Pixelpiper

Redump 1G1R Collection Tool - Disc Systems: PS + PSP, XBOX, GC, Dreamcast, etc.

Recommended Posts

Posted (edited)

DCTF: Disc Collection Title Filter

I've made a web-based tool to, among other things, help make 1G1R sets for any disc-based (redump) platform and it needs some final testing before I give it a wider audience. It should work on any current browser on any computer/OS. Tested with Chrome, Safari and Firefox on MacOS, Windows and iPhone (use it on a computer). Please let me know if you can give it a look. Bonus if you have a Redump account.

https://salumba.com/romlist

The tool can help make bulk download lists that can be imported into download managers, or download single titles. It can also make batch command files that can be used at a command prompt to manipulate your current set of titles - like to clean a huge mess that might be left after downloading every game from every region for a platform.

Title selection goes through a de-duplification and priority system which looks at the base title, regions, version #'s, edition type and languages before finally applying a manually-created clone filter with North American or European preference (or it can be disabled).

The result of the final list is that every single title will be displayed restricted by your filtering choices. Want only US titles? Want titles from any region as long as they support English? Want to prioritize the different macro-regions when selecting titles? Want only US and European but no Japan/Asia regardless of language? Want only titles released in North America that also feature French language? Want titles from anywhere in the world that have both English and German?

All of this and much more can be done.

In addition to pulling only the titles matching your settings, this tool will construct appropriate filenames to match the No-Intro and Redump standards. You can add any file extension you want to the displayed filename to match the downloads available to you - for example you might need files labeled ".zip" for one platform/site and ".rar" or ".7z" elsewhere.

Quick overview:

Pre-Filter & Restrict the Source Data:

  • The tool builds a master list of titles according to the options you set
  • Basic setting (mandatory) System/ Platform - supports Nintendo GameCube, Wii, Wii-U, Sony PSX/1-PS4 & PSP, XBOX, XB360, XBOne, Sega CD, Saturn & Dreamcast, PC, etc.
  • Release Category (optional) - All Categories,  Games, Applications, Educational, etc.
  • Region (optional) - All Regions, Americas, Europe, ASIA, USA, Canada, Japan, Germany and more.
  • Language (optional) - All Languages, English, Japanese, French, German, Spanish, Italian, Portuguese
  • Hide selected "Edition" types: Demos, Previews, Beta or Unlicensed (tool starts with defaults pre-selected)

Final filtering Options:

  • Language - huge list - you can set the data source to All or a different language if you want to force 2 languages on every title
  • Regional priority of 3 top-level (macro) regions covering all releases allowing to block one of the three
  • Clone lists to remove duplicates that have different titles from different regions (pick North American or European preference)
  • Exclude any number of keywords or phrases you specify (example: you don't want to list anything that contains "tetris")
  • Only include keywords of phrases you specify (example: you want to only list "Harry Potter" titles)
  • Keywords can be match as whole words/phrases or as substring (example: "bat" won't match "Batman" for Whole Word but will for  Substring)
  • Import/upload your own file list (dir /b  to a text file in Windows to a or cut-and-paste to a text file from Finder in MacOS) to exclude from results
  • Import/upload your own "Have" title list made with ClrMamePro to exclude from the results
  • Uploaded file should contain filenames in No-Intro/Redump format with or without any file extension
  • Uploaded file is temporarily saved to the server with unique encrypted filename, overwritten when you process with new options and deleted every 10 minutes

De-Duplification:

  • Based on title matching
  • Based on clone list (built-in and platform automatically selected)
  • Versioning (higher versions have preference)
  • Editions (Originals have preference)
  • Regions (multiple region hierarchies automatically selected by your macro region priority setting)
  • Languages (always contains the languages you selected, titles with fewer languages have higher priority)

File naming: No-Intro standard

  • "The" properly moved after base title
  • Region inserted into the correct place
  • Disc#'s correctly placed
  • Languages only listed when more than 1 and in the preferred order
  • Version # is optional - always omitted when "v1.0" - also supports "Rev #" style
  • File extension is optional and can be set by you

Custom formatting:

  • Specify any text to include before filenames
  •   • URL text to make download links: "http://somesite.com/files/" - links can be copied/imported to download managers like JDownloader
  •   • Shell/command text for batch files: "mv "  (for example to move the files in a *unix shell)
  • Specify any text to include after filenames
  •   • Shell/command path/arguments: " /volumes/somedrive/files_to_keep/"  (used with a move this would move all the files to this folder)

Final (personal) lists also temporarily saved on the server so that you can download them directly instead of copy-paste

  • Saved with unique & encrypted names so multiple people's lists will never interfere with each other & never be cached
  • HTML list link can be copied with "copy link" and that can be pasted directly into JDownloader or other download manager
  • Link files automatically deleted within about 10 minutes

 

Screenshot:image.thumb.png.d75e64565d62ca0c3a16fabd4b744d25.png

 

Edited by Pixelpiper
  • Like 5

Share this post


Link to post
Share on other sites
Posted (edited)

OK, finished editing the first post with some details.  If you can help take a look, let me know.

A couple of Q & A:

Q: Can this tool list English games from any region?

A: Yes

Q: What about English games from Europe or Japan or... that aren't marked English in the filename?

A: Yes

 

Edited by Pixelpiper
  • Like 2

Share this post


Link to post
Share on other sites
Posted (edited)

Feature complete and working well.

Next tool I'm going to work on:  Full download maker.  Goes like this:

1. Pick platform from a pre-defined list

2. Select  "Auto List with options" - the new tool will build a game list for you with basic options

or "Filter from manual list" - Paste a list from the filter tool in this thread or from clrmamepro "missing" or any kind of game list with or without regions - title only or full filename

3. Set download source option to AUTOMATIC or MANUAL (Paste a list of one or more download pages - it doesn't matter if it's 20 or 30 or 500)

4. press GO.

The tool will spit back full HTML download links for every title it could find matching your list or matching the automatic list it generated using your preference (like all English titles for this region and that region).  Toss that HTML file into a batch downloading app and presto, all the games are downloaded. The beauty here is that for every list, the download could be coming from any number of sources - it doesn't matter and it means you can grab as few or as many regions as you want - the filename formats at the different download sites can also be completely different - it simply doesn't matter, you'll still get all the titles you selected.

Edited by Pixelpiper
  • Like 2

Share this post


Link to post
Share on other sites

Sounds interesting. Can you post a video to see it in action and understand it better

Share this post


Link to post
Share on other sites

Sounds great btw. If there's a public repo, I'd love to have a look a the code. (Curiosity)

Share this post


Link to post
Share on other sites

@PixelpiperLike others, I'm curious and would like to use this. I came across this thread searching for "Redump 1G1R". There's not many options (if any at all) for non No-Intro sets. 

I've signed up on this forum just to follow this thread. I look forward to updates.

Share this post


Link to post
Share on other sites
20 hours ago, zetec-s-joe said:

If implemented TOSEC and No-intro it would be the all in one of making upto date 1G1R sets :)

I second this. There are no tools for building 1G1R sets for inexperienced users.

Share this post


Link to post
Share on other sites

All in one app for everyone would be amazing, it would also pretty much end the need forthe outdated HyperList XML files as you could make your own current sets using this one tool

Share this post


Link to post
Share on other sites
Posted (edited)

Oops, I missed all these messages. Guess I forgot to subscribe to my own topic.    Here's a link to test it out:

https://salumba.com/romlist

I have some enhancements planned, including a massive expansion of the built-in clone lists. Certain "newer" platforms currently won't show any data because the source of that data is hidden behind a Redump account login. I've asked for an account and submitted a dump for consideration, but who knows - I can probably use some help here from someone with an account.

The most important thing to know before using my tool is that you should forget or ignore everything you know about other ROM tools.  Forget DATs, forget clrmamepro, etc. Just read the descriptions and post in here to let me know what needs explaining. Looking to simplify and make it as clear as possible - I can eventually add help pop-ups too.

Edited by Pixelpiper

Share this post


Link to post
Share on other sites
On 4/20/2020 at 6:13 AM, zetec-s-joe said:

If implemented TOSEC and No-intro it would be the all in one of making upto date 1G1R sets :)

There's a great 1G1R tool already for No-Intro sets that uses P/Clone DAT files. It requires that you've already downloaded all the ROMS in advance, but No-Intro cart-based stuff is miniscule compared to disc-based so it's not an issue, IMO. I think the biggest hold-back on the tool is that it's native Python and you have to run it from a command-line with Python set up. I hope to help the author get around that in the future. Andre, the author, is working on a complete and more feature-rich re-write at the moment: https://github.com/andrebrait/1g1r-romset-generator

 

  • Like 1

Share this post


Link to post
Share on other sites

Ahh fair enough, You done an amazing job on your version and so easy to use. It would just be really cool to have all in one. Great job and thank you again

Share this post


Link to post
Share on other sites
Posted (edited)

Once I have the rest of the features and data in place for disk systems, I'll consider adding cart systems too. The concept of my tool has always been to prepare the 1G1R list BEFORE downloading anything - so I didn't start with cart systems because those downloads were already pretty small. ;)

I can definitely leverage info already found in DATs and elsewhere for those systems. The beauty of the Redump data however is that it contains a trove of information that's left out of all DATs, including the language of single-language games - that alone is huge on some platforms.

Edited by Pixelpiper
  • The Cake is a Lie 1

Share this post


Link to post
Share on other sites

is there any way to verify the file exists on the url provided? is there any way to add multiple urls eg sometimes regions are split between urls and provide a list of titles not found?

Share this post


Link to post
Share on other sites
9 hours ago, Newlander said:

is there any way to verify the file exists on the url provided? is there any way to add multiple urls eg sometimes regions are split between urls and provide a list of titles not found?

That's what the second tool will be for. ;)  Any text/URL on this tool is strictly for text/html output - it doesn't visit those links.

In the meantime, you can generate the list multiple times, each with a different URL and make sure your download manager can detect duplicates, so it will ever only grab each file once.

Share this post


Link to post
Share on other sites
14 hours ago, Pixelpiper said:

Once I have the rest of the features and data in place for disk systems, I'll consider adding cart systems too. The concept of my tool has always been to prepare the 1G1R list BEFORE downloading anything - so I didn't start with cart systems because those downloads were already pretty small. ;)

I can definitely leverage info already found in DATs and elsewhere for those systems. The beauty of the Redump data however is that it contains a trove of information that's left out of all DATs, including the language of single-language games - that alone is huge on some platforms.

Great job man (Y)

Share this post


Link to post
Share on other sites
Posted (edited)
On 4/6/2020 at 5:13 PM, Pixelpiper said:
  • Language - huge list - you can set the data source to All or a different language if you want to force 2 languages on every title

I am not sure to understand properly this feature.

Is it possible to have all games released in French and English languages (French version is prefered if exists) ?

Great tool btw :)

Edited by T2Fr

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...