ZIP files: Supported or not supported?

[Slag-O-Matic]

I've twice now run into the same situation and I'm wondering if it's something I'm not understanding or what.

I have a sizeable Dreamcast library. Each Dreamcast title is self-contained in a ZIP file which, if expanded, results in a folder containing a CUE file and several BIN files. I was able to successfully import this library into LaunchBox without incident. It pulled down all the happy metadata and everything, and life seemed good. But when I went to play a title -- ANY title -- I would get the initial launch screen and then nothing. It just reverted back to the title page.

I am using the Flycast core. According to this page:

https://docs.libretro.com/library/flycast/

...Flycast supports ZIP files. However, after a bit of experimentation I discovered that unzipping the file into its composite folder, and subsequently pointing the game towards the CUE file, would make the game play properly.

Armed with this knowledge I unzipped all of my games into their component folders, removed the Dreamcast console from LaunchBox (keeping the metadata intact), then reimported the newly-unzipped Dreamcast library. When it was finished it SEEMED like everything went well, but I had the same problem when I tried to start a game: I would get the initial launch screen and then nothing.

Further investigation revealed that the second import process pointed to the first BIN file in each folder, not the CUE file. I corrected one entry and tested it successfully.

HOWEVER, I am left with three questions.

1. Does Flycast, or does it not, support ZIP files? If yes, then (a) why isn't it supporting MY ZIP files and (b) how can I get it to support my ZIP files?
2. Since it needs the CUE file to play, why is its default action to point to an unplayable BIN file rather than the playable CUE file? And how can I stop it from doing that going forward?
3. Without painfully selecting each and every game and editing it (from BIN to CUE), how can I "re-index" the existing XML file to point to the CUE files? I tried a little search-and-replace action but quickly discovered it wouldn't work since (as I understand it) each game entry lists the "launch file" (what should be the CUE file but is now the BIN file) and what I'm assuming are "associated files" which are the BIN files. So the first BIN file is listed twice -- once as the "launch file" and again as an "associated file". A simple search and replace (find " (Track 01).bin" replace with ".cue") would rename not only the "launch file" but ALL of the associated files, so it would still break. How can this be fixed in bulk?  (EDIT: Strike this one, figured it out.)

Edited July 8, 2022 by Slag-O-Matic

[Headrush69]

You have to remember that the flycast core supports naomi games as well, not just Dreamcast.

Naomi roms are indeed zipped and the flycast core  itself supports reading from those. The flycast core itself doesn't support manually zipped cue/bin files. (I don't believe)

LB also supports unzipping archives, but it does this and than passes those files to the emulator cores. In this case the core doesn't know which file to use/ (bin/cue)

I don't think that it's a default action, but more likely an alphabetic ordering. Try renaming you cue file to something alphabetically before the bin file (but point to the same bin) and see if it loads now).

I would either:

1. Import your Dreamcast cue files manually
2. Convert your cue/bin to CHDs.

I've gone with 2. It saves valuable space and works flawlessly.

 

Edited July 9, 2022 by Headrush69

[Slag-O-Matic]

13 hours ago, Headrush69 said:

I've gone with 2. It saves valuable space and works flawlessly.

I like this idea. Trying it now. Thanks!

EDIT: So I found and downloaded CHDMAN and converted one of my folders into a CHD file (using the "CUE or GDI to CHD" script). It took a few minutes but was successful. I pointed LaunchBox to the newly-created CHD file but now I'm at the same point as I was before with the ZIP file. I launch the game, it says "NOW LOADING", then a couple of seconds later it just stops and goes back to the game entry. Any ideas?

Edited July 9, 2022 by Slag-O-Matic

[Headrush69]

31 minutes ago, Slag-O-Matic said:

I launch the game, it says "NOW LOADING", then a couple of seconds later it just stops and goes back to the game entry. Any ideas?

Do you still have the extract ROM archives before running option enabled in the emulator settings?

[Slag-O-Matic]

1 hour ago, Headrush69 said:

Do you still have the extract ROM archives before running option enabled in the emulator settings?

I just spent the better part of 20 minutes looking for this option in LaunchBox and RetroArch and failed. Where is this option?

[neil9000]

30 minutes ago, Slag-O-Matic said:

I just spent the better part of 20 minutes looking for this option in LaunchBox and RetroArch and failed. Where is this option?

Tools/Manage/Manage Emulators/Name Of Emulator, uncheck the box.

[k_rollo]

23 hours ago, Slag-O-Matic said:

I like this idea. Trying it now. Thanks!

EDIT: So I found and downloaded CHDMAN and converted one of my folders into a CHD file (using the "CUE or GDI to CHD" script). It took a few minutes but was successful. I pointed LaunchBox to the newly-created CHD file but now I'm at the same point as I was before with the ZIP file. I launch the game, it says "NOW LOADING", then a couple of seconds later it just stops and goes back to the game entry. Any ideas?

Last time I checked, converting the CUE file specifically for Dreamcast ROMs does not work. You need to convert the GDI file. I got Dreamcast ROMS in BIN/GDI instead and the CHD conversion worked. If your ROM download comes wth both CUE and GDI (like it does from Vimm's Lair), delete the CUE files first before running the .bat script.

[neil9000]

1 minute ago, k_rollo said:

Last time I checked, converting the CUE file specifically for Dreamcast ROMs does not work. You need to convert the GDI file. I got Dreamcast ROMS in BIN/GDI instead and the CHD conversion worked. If your ROM download comes wth both CUE and GDI (like it does from Vimm's Lair), delete the CUE files first before running the .bat script.

That will just depend on what file extensions are in the bat file you used, you can easily edit it to just add more extensions.

[k_rollo]

14 hours ago, neil9000 said:

That will just depend on what file extensions are in the bat file you used, you can easily edit it to just add more extensions.

OP particularly said he used the "CUE or GDI to CHD" and that specfic script is popularly downloaded from the internet along with chdman binaries. He can delete the CUE or edit the script to remove CUE, whichever way he is comfortable. Point being, you don't want chdman using the CUE instead of the GDI.

Edited July 11, 2022 by k_rollo

[Slag-O-Matic]

Okay, so this is getting a little exhausting.  😕

I tried converting the Dreamcast games to CHD and as k_rollo seems to have predicted, they don't work. I don't even get the "NOW LOADING" screen. You hit PLAY and just...nothing happens. Doesn't matter if "extract ROM archives" is enabled or not. However, I looked at a bunch of past posts on these forums and a lot of people seem to have no problems using CHD files with Dreamcast so I'm not sure what's different in my setup.

I was happy to see a substantial size reduction in doing the CHD conversion, plus I wanted to make sure the conversion was actually working and not breaking games, so I converted my TurboGrafx-CD library (which was previously a bunch of folders) to CHD and they all work nicely, with about a 40% space savings. So, win-win, plus proof that the conversion wasn't breaking anything.

I then went back to my original zipped files of my Dreamcast library. Using the "extract ROM archives" function I can play the games in ZIP format (which is good) but there's a pretty substantial lag whenever any game is loading (which is bad). I get that this is from the system unzipping the game into a (temporary?) location when loading, but in the interest of not waiting 30+ seconds for a game to unzip I'd like to eliminate that lag. Any ideas why my emulator won't use CHD files? If I can get the Dreamcast games working in CHD format that would be better as there doesn't seem to be much of any lag there, while still playing all the titles. Or MUST they be GDI files? Are GDI files similarly compressed (i.e., is there a space savings going from a folder-of-files to a GDI file, like there is when going to a CHD file)? Is there a way to get my emulator working with CHD files? Or else, a way to convert my folders-of-files library to GDI files (even if I have to go to CHD first)?

[k_rollo]

  

1 hour ago, Slag-O-Matic said:

I tried converting the Dreamcast games to CHD and as k_rollo seems to have predicted, they don't work. I don't even get the "NOW LOADING" screen. You hit PLAY and just...nothing happens.

15 hours ago, k_rollo said:

Last time I checked, converting the CUE file specifically for Dreamcast ROMs does not work. You need to convert the GDI file. I got Dreamcast ROMS in BIN/GDI instead and the CHD conversion worked. If your ROM download comes wth both CUE and GDI (like it does from Vimm's Lair), delete the CUE files first before running the .bat script.

Did you follow these as instructed? I assume you still converted the CUE? As mentioned, conversion from CUE to CHD in Dreamcast does not work. The ROMs must be in BIN/GDI before conversion.

 

1 hour ago, Slag-O-Matic said:

Are GDI files similarly compressed (i.e., is there a space savings going from a folder-of-files to a GDI file, like there is when going to a CHD file)? Is there a way to get my emulator working with CHD files? Or else, a way to convert my folders-of-files library to GDI files (even if I have to go to CHD first)?

CUE and GDI are not compression formats. They are nothing but text files and can be opened with Notepad. The files list the sequence of tracks in a disc media.

[neil9000]

7 hours ago, Slag-O-Matic said:

Okay, so this is getting a little exhausting.  😕

I tried converting the Dreamcast games to CHD and as k_rollo seems to have predicted, they don't work. I don't even get the "NOW LOADING" screen. You hit PLAY and just...nothing happens. Doesn't matter if "extract ROM archives" is enabled or not. However, I looked at a bunch of past posts on these forums and a lot of people seem to have no problems using CHD files with Dreamcast so I'm not sure what's different in my setup.

I was happy to see a substantial size reduction in doing the CHD conversion, plus I wanted to make sure the conversion was actually working and not breaking games, so I converted my TurboGrafx-CD library (which was previously a bunch of folders) to CHD and they all work nicely, with about a 40% space savings. So, win-win, plus proof that the conversion wasn't breaking anything.

I then went back to my original zipped files of my Dreamcast library. Using the "extract ROM archives" function I can play the games in ZIP format (which is good) but there's a pretty substantial lag whenever any game is loading (which is bad). I get that this is from the system unzipping the game into a (temporary?) location when loading, but in the interest of not waiting 30+ seconds for a game to unzip I'd like to eliminate that lag. Any ideas why my emulator won't use CHD files? If I can get the Dreamcast games working in CHD format that would be better as there doesn't seem to be much of any lag there, while still playing all the titles. Or MUST they be GDI files? Are GDI files similarly compressed (i.e., is there a space savings going from a folder-of-files to a GDI file, like there is when going to a CHD file)? Is there a way to get my emulator working with CHD files? Or else, a way to convert my folders-of-files library to GDI files (even if I have to go to CHD first)?

Have you pointed Launchbox to the new files? Do the CHD's work just in the emulator alone.

5 hours ago, k_rollo said:

Did you follow these as instructed? I assume you still converted the CUE? As mentioned, conversion from CUE to CHD in Dreamcast does not work. The ROMs must be in BIN/GDI before conversion.

Again, like i previously said the cue will convert just fine, as long as the batch file you have contains cue as a source to convert.

You can add any file type to the highlighted part, i used the same script to convert cue/bin PS1 games with no issues, the batch file just needs to know what files it is looking for, i could add gdi or cdi to the above highlighted part of the batch script and it would then convert those too.

[Headrush69]

7 hours ago, Slag-O-Matic said:

Any ideas why my emulator won't use CHD files? If I can get the Dreamcast games working in CHD format that would be better as there doesn't seem to be much of any lag there, while still playing all the titles.

Are you saying none of the Dreamcast converted CHDs work, or just some?

The fact is there is a lot of poorly ripped Dreamcast discs. Many times the contents were modified to fit a smaller sized CD. Often .cdi files were just renamed to .gdi even though they weren't properly ripped.

Convert the ones you can, and for the ones that don't the easiest solution is just to look for a more reliable source and reacquire.

I ran into the same issues but now have my small (80) Dreamcast library converted and all work perfectly.

P.S. I would stay away from the "extract ROM archives" option. In most cases it's unlikely needed and doesn't have anything to do with Retroarch's ability to load zip archives. You might also want to make sure that option isn't still enabled on the associated platforms section of the emulator as well as the main emulator settings screen.

[k_rollo]

3 hours ago, neil9000 said:

Again, like i previously said the cue will convert just fine, as long as the batch file you have contains cue as a source to convert.

You can add any file type to the highlighted part, i used the same script to convert cue/bin PS1 games with no issues, the batch file just needs to know what files it is looking for, i could add gdi or cdi to the above highlighted part of the batch script and it would then convert those too.

CUE will convert fine on PS1, NOT on Dreamcast specifically as I mentioned before. People assume PS1 BIN/CUE and Dreamcast BIN/CUE behave the same in CHD conversion (which I also assumed previously), but they don't. Don't ask me why, I just know that reddit post saved me a lot of trouble haha.

He already tried converting the CUE files anyway and failed as I had predicted so that supports the theory. He can keep repeating the same process that failed or try a different solution that I can confirm also worked for me when I had this problem.

Good luck to OP!

Edited July 11, 2022 by k_rollo

[Glenn72]

GDI to CHD is really the way to go for all Dreamcast games.  The only CDI files that I have are a few fan made compilations.

GDI are the full source files which can be up to 1gb while most CDI files contain down sampled audio and video so they could physically fit on a 700mb cd.

As we no longer have 700mb space limitations with Emulation, GDI files converted to CHD provides lossless compression.

[Slag-O-Matic]

Well, this is disappointing. At the very least I'm getting an education. Thanks all for explaining things to me. I'm getting there.  🙂

First, to answer the question posed to me: None of my CHD files work; k_rollo is absolutely correct when he says that a conversion from BIN/CUE to CHD does not work with Dreamcast games. I converted about a hundred titles and none of them work when converted to CHD.

So, for everyone's reference, my starting point is an ~800GB Dreamcast library which consists of about 1400 ZIP files.

If I point LaunchBox at the ZIP files and enable the "extract ROM archives" option, the games work. However, it takes upwards of 45 seconds for any game to load as the system has to decompress the ZIP file before it can be played. This is not optimal.

If I unzip all the ZIP files I get a folder full of about 1400 subfolders (one for every game/ZIP file) with each subfolder containing a CUE file and multiple BIN files. If I point LaunchBox at this folder it doesn't work but only because LaunchBox is looking at the first BIN file in each subfolder. But if I do a bit of search-and-replace on the XML file I can point LaunchBox to the appropriate CUE file in each subfolder and then the games work. The downside here is the nearly 100% increase in drive space required to store the library.

So if I'm understanding what y'all are saying here, my entire library is effectively shite and I need to redownload a new library in GDI format. I could then convert those GDI files to CHD files which should play fine, have no loading lag like the ZIP files have, and be smaller than the unzipped files. Is that correct?

I don't suppose there's a way to convert the CUE file to a GDI file, is there? I know it's not as simple as changing the file extension (yeah, I tried) but my alternative is to throw away about two weeks of downloading which I'd like to avoid if possible.

 

[neil9000]

20 minutes ago, Slag-O-Matic said:

If I point LaunchBox at this folder it doesn't work but only because LaunchBox is looking at the first BIN file in each subfolder.

Go to that folder in Windoes, put *.cue in the window search and drag and drop the results to Launchbox to start the import wizard with the cue files.

[Headrush69]

1 hour ago, Slag-O-Matic said:

k_rollo is absolutely correct when he says that a conversion from BIN/CUE to CHD does not work with Dreamcast games. I converted about a hundred titles and none of them work when converted to CHD.

Might be the quality of the cue file. (Some may have better details that chdman needs)

I just download a new Dreamcast game in CUE/BIN format, and chdman converted it to CHD just fine and it plays without issue.

1 hour ago, Slag-O-Matic said:

So if I'm understanding what y'all are saying here, my entire library is effectively shite and I need to redownload a new library in GDI format. I could then convert those GDI files to CHD files which should play fine, have no loading lag like the ZIP files have, and be smaller than the unzipped files. Is that correct?

Can't speak for the status of your library. but you are correct with the rest. CHD is the way to go.

[k_rollo]

7 hours ago, Slag-O-Matic said:

First, to answer the question posed to me: None of my CHD files work; k_rollo is absolutely correct when he says that a conversion from BIN/CUE to CHD does not work with Dreamcast games. I converted about a hundred titles and none of them work when converted to CHD.

Thanks for confirming, I knew that was the problem.  I also think it's not "faulty" CUEs either, because they work when you pointed LB to the CUE in the XML, so that tells us your CUE files do work but just don't convert.

7 hours ago, Slag-O-Matic said:

So if I'm understanding what y'all are saying here, my entire library is effectively shite and I need to redownload a new library in GDI format. I could then convert those GDI files to CHD files which should play fine, have no loading lag like the ZIP files have, and be smaller than the unzipped files. Is that correct?

This is correct. The GDI to CHD conversion is everything you want and what I am also enjoying now. I can even convert fanmade Dreamcast ports such as Dolphin Blue from GDI to CHD. I wouldn't say your library is completely sh*te though. You effectively solved your original problem with the XML solution, but you just had an additional requirement of converting to CHD upon your discovery of the format. I don't blame ya.  In my case however, it was faster to re-download the GDI because I don't exactly take entire romsets making them easier to manage on mishaps (such as this).

7 hours ago, Slag-O-Matic said:

I don't suppose there's a way to convert the CUE file to a GDI file, is there? I know it's not as simple as changing the file extension (yeah, I tried) but my alternative is to throw away about two weeks of downloading which I'd like to avoid if possible.

You are now struggling with the "Sunken Cost Fallacy" where you are reluctant to abandon a strategy because of an initial heavy investment on it (i.e. 2 weeks worth of download), which I don't blame ya for either. However, this can keep you stuck.

As I see it, you have these options:

1. Pick only the games you like when re-downloading GDI rather than the whole romset. You give up the "completeness" factor. Results are certain.
2. Retain BIN/CUE format and point LB to CUE as you already did. You give up CHD compression. Results are certain.
3. Keep looking for a way to batch convert CUE to GDI (I haven't seen a method that worked). You get completeness and CHD compression. Results are uncertain until you convert CUE to GDI first.
4. Re-download romset directly in CHD format if available. You repeat the download process (to a certain extent). Results are certain.

Personally, I picked #1 when I had this problem. Not only did it save me a lot of time but also even much bigger space in addition to the CHD conversion. Time and space were more valuable to me than completeness. Hope this helps you decide, it will be up to you.

Also, I'm a girl haha.  Good luck OP!

Edited July 16, 2022 by k_rollo
Added #4 as mentioned by Suhrvivor

[Suhrvivor]

Instead of getting GDIs and then convert them yourself to CHD, you could just get the CHD instead (they're probably in the same place you get GDIs).