ExoDOS Import

[harryoke]

25 minutes ago, cleverest said:

Awesome! You should also create an angelic theme'd one for users on the light side though...

Will do , good idea

[eXo]

2 hours ago, Dan said:

Great news eXo. Worth doing it right with a project such as this. Wonder if you can get anyone to help. I would have back in the day when I had more time. They should have a stamp “certified by exo” on the games DB so that we know which items have the correct data  

That is actually a concern of mine. At one point I fixed a ton of Sega entries, and then someone came along later and tried to revert a lot of them. I don't want to bother tweaking 7,000 DOS games profiles to "perfection" just to have someone come along and screw it all up later.... so I'm reluctant to put too much time into the games db from my end.

2 hours ago, FistyDollars said:

@eXo I've been working on this off and on (as you can relate, it's a very time consuming process.) I've added every game from the collection right up to the letter H, it's why if you go to the database it's a bit "front of the alphabet heavy." I was going at a pretty good pace for a while but life kind of gets in the way. I've since been back on board , going as fast as I can. So the good news is that it's already being done, it's just going to take me a little more time.

Also, I'm going off of the games in the 2.0 collection, so if there are any major changes, they'd need to be added as well.

Please let me know if the entries at the front of the alphabet are up to snuff, I've worked pretty hard on them (including submitting the clear logos from Hyperspin when appropriate.) Also, I've made sure that all the artwork isn't watermarked.

Interesting, quite a few games don't match, but on the other hand, several games have matched that I knew had come from eXoDOS. SO I figured someone had been busy at least. In the Adventure collection I have around 1,100 games currently, and I got about 450 matches when I ran the search. Of those, I have found about 20 or so that matched incorrectly and totally changed the game into something unrelated. I have found another 20 or so that got various versions of a game confused and it decided to try and and homogenize all of them to one version (Mixed-Up Mother Goose and it's 4 friggin different versions comes to mind, along with all of the VGA Sierra remakes & fanmakes).

 

I'll check later today and see about giving you a list of the Adventure games from A-H that don't match. That way you can determine if its because they are additions to 3.0 or they just aren't matching properly for some reason. I am noticing that many of the games that match don't have their genre set properly. Very few, if any, of the inter-active fiction games are bringing that genre over with them. So I've been manually adding that to all of the text adventures. And then for all of the ascii based games (think more action oriented, like ZZT or toher top-down text based graphics) I've been setting them as "Text-Based".  I also find many of them have no year set, so I'm having to add a release year as well. I'm not sure if this is an issue with the games on the gamesdb, or if there was a problem with the sync, or if it is something else. But those are the main issues.

1 hour ago, harryoke said:

Quick update on Exodos/win3XO Bigbox Startup video, designed logo .... went for the trusted evil skull & fire .... hope u like

Wow, fun stuff! Great work. I'm partial to retro stuff myself, but this is pretty cool. Thanks for doing this.

[FistyDollars]

I'm wondering if there are issues translating the genres from eXoDOS to the Games Database. They're not exactly 1 to 1, as there are only a set list of genres available in the database, and a couple of them were only recently added.

Currently the genres are:

• Action
• Adventure
• Beat 'em Up
• Board Game
• Casino
• Construction and Management Simulation
• Education
• Fighting
• Flight Simulator
• Horror
• Life Simulation
• MMO
• Music
• Party
• Platform
• Puzzle
• Quiz
• Racing
• Role-Playing
• Sandbox
• Shooter
• Sports
• Stealth
• Strategy
• Vehicle Simulation
• Visual Novel

That might be why certain things weren't matching, especially Interactive Fiction. If you talk to @Jason Carr or @Vlansix I'm sure they could tweak the genres to help you out.

[harryoke]

Art is so much fun

@eXo @cleverest

[spektor56]

@eXo Do you already have all the metadata?  I can see if i can work something out getting it into the database if i get some free time.  Have a baby on the way due 2 weeks from now and i'm not sure how much time out of my day i will have to do other things lol.  Pretty sure someone made a utility a while ago as well to do mass uploads to the database.

Edited December 1, 2017 by spektor56

[eXo]

Yes, I have all my game data in a database. I'll ask Alex about any potential to do a mass upload. He will know if there is anything to do this. The key is getting the art in as well though. Otherwise a db sync to those games will blank out my existing art.

[harryoke]

@eXo hope you like it

[eXo]

On 12/1/2017 at 11:10 AM, FistyDollars said:

@eXo I've been working on this off and on (as you can relate, it's a very time consuming process.) I've added every game from the collection right up to the letter H, it's why if you go to the database it's a bit "front of the alphabet heavy." I was going at a pretty good pace for a while but life kind of gets in the way. I've since been back on board , going as fast as I can. So the good news is that it's already being done, it's just going to take me a little more time.

Also, I'm going off of the games in the 2.0 collection, so if there are any major changes, they'd need to be added as well.

Please let me know if the entries at the front of the alphabet are up to snuff, I've worked pretty hard on them (including submitting the clear logos from Hyperspin when appropriate.) Also, I've made sure that all the artwork isn't watermarked.

As I've been working through this more, it is very apparent how much time you've put into adding games. I really can't thank you enough for all the effort you have put into it. I know first hand how tedious it can be.

[eXo]

This morning I finished a week long process of going through and making sure every game was integrated into launchbox correctly. Essentially, I went through every single game and did the following:

• If it was linked to the gamesdb, I made sure it was linked to the correct game (the auto scan linked dozens of incorrect titles)
• fixed genres to match my categories (adding 'Interactive Fiction', 'Horror', etc....)
• added all game title and gameplay screens shots from my collection
• added all extras as "external applications". This allows you to right click on a game and access any maps, copy protection, etc...
• verified all manuals were linked properly
• added year/date info to every game
• fixed a ton of bad info coming out of the gamesdb
• normalized all developer/publisher information (so you don't have Sierra, Sierra Online, Sierra On-Line, and Sierra On-Line, Inc.)

It was tedious, but makes for a much better experience in LaunchBox. Once I release this however I will warn folks though, update from the gamesdb at your own peril. At some point the gamesdb will be accurate and a great way to get this information. FistyDollars has gone a long way in helping towards that goal. Unfortunately, as it stands, there are many entries in the gamesdb that are totally pointless. Just the games name, no metadata, no cover art, nothing.  So if you were to update one of those games, it would essentially blank out all the data I have added.

That said, the way I am distributing this collection will have a zipped copy of LaunchBox inside. So if someone does screw it all up and realizes too late, simply running my setup file again will set things back to the way they were when I released the pack.

 

As I am packing this collection up in preparation for release, it brings up other challenges though. Such as, when I release the next pack, RPG games, how will a person merge them together. As I am sure you know, LB uses xml files for each platform. It seems to me the best way to do this would be to have a handy way to merge two MS-DOS.xml files together. Normally this would be as simple as taking all of the <games></games> from one and pasting them between the LaunchBox tags in the other. However the use of "AdditionalApplications" to access the extras seems to create a new section below the games tags. I'm not sure if these *have* to be at the bottom (probably not), but it's something I'm keeping in mind. 

So, the reason I'm mentioning this is to see if anyone is interested in pulling together a small xml merge app for me. I haven't spent a lot of time deciding the best way to execute this during initial setup, as I figured I would talk through it with whoever was involved. 

I will likely release the Adventure pack before any such tool is created, and use the time I am working on the RPG pack to try and get it all worked out. Hope someone pipes up. However if no one has time to help, I will take the lowtech route and simply provide "pre-merged" xml files for folks to drop in if they have multiple packs.

Edited December 6, 2017 by eXo

[NLS]

eXo thank you for your FANTASTIC work.

It makes me almost sad I was never really an MS-DOS gamer (MS-DOS gamer sounds like a joke coming from Amiga).

You are helping LB become even better, the Go To solution for any retro gaming needs (and even non retro as you can use it for Steam and normal installs too). Hope LB team realizes that (I think they do).

Now for the XML thing and until an automatic solution comes up (if ever)...

I have three different ideas.

1) The worse idea. Make it separate platforms. So MS-DOS_RPG is one. I hate it but is a solution.

2) You can release separate versions of the XML.
In the download package you will have the released genre's XML. So when you release RPG, it will be just that.
Outside the package and in any description you make (in PD, or wherever) you also post separately a new XML that includes ALL released genres. So if you released RPG first, you only have the XML that includes RPG games. If you release Action second, then in the package you have XML with only Action package and in the post talking about the package, you post an XML with already merged RPG and Action (and in separate genres/categories). With the third release you do the same... only include the single one in the package and a complete (three genres) outside.

3) Similar to #2 but with more work. Since you already know your complete contents and you only need the clean up (and the actual content upload), you can already make the COMPLETE XML, but fill the non-released (and non-cleaned-up) genres/games with simple placeholders! This would be the best, as it will make known even to people that don't have the other (released or unreleased) parts that they need to get them (or wait for them)!
So it will be a complete list, with normal genre separation but if you only released RPG and someone scrolls to an Action game, it will instead get a placeholder picture and placeholder description (copy paste) that says "this belongs to the Action part of my collection, expected release so and so, please if interested look for it so and so. Make sure to replace MS-DOS.xml with the latest version found so and so...".
This would be great, but needs more work now (and less later).

 

[eXo]

Option 1 is a no-go.  A unified MS-DOS platform for all of my games is important to me. You are right however in that while being the "worst" solution, it would also be the simplest. I think I would get crucified at the emulation altar if I went and modified the game platforms to something in the realm of genre. I can already see various forums burning exo effigies.

Option 2 is the "dirty" solution that I am going to plan on using as of now unless a cleaner "app" solution presents itself.

Option 3 is a clever idea. However I don't want to force anyone to *have* to download all 5 volumes. I know from the various feedback that I receive that there are many people who just grab one or 2 sets. I can look at the number of downloads for each torrent and see some (like Action) have been downloaded thousands more times than the Simulation pack. 

Edited December 6, 2017 by eXo

[StammesOpfer]

Seems like it would be easy enough to use a batch file to merge a couple different xml files as long as you don't have to dedup them too. I can try and run with this if no one wants to create an actual executable.

[eXo]

There isn't duplicate data. However, the batch would seemingly have to be smart enough to determine the section that has games and the section that has applications and paste them separately into different sections.

[StammesOpfer]

I think I could do it with a couple loops. Iterate through until it see a specific tag. Output to 2 separate files until you have read all source files then merge the 2 temp docs.

Edited December 6, 2017 by StammesOpfer

[eXo]

That would probably work. 

I think my largest concern with going down this path is finding a way to do this so the end user isn't involved.  

Generally, people like to seed from the same source they access the games from, as not everyone has an extra 500gb laying around to dedicate to seeding this. That means, to have a complete set, all 5 volumes have to reside in the same directory. Since only one MS-DOS.xml file can exist in that folder at any given time, that blows up.

So maybe the solution is to have a folder with the xmls in it. Maybe something like ADV.xml, RPG.xml, etc...  Then, during the setup process, your bat kicks off and checks this folder and creates a merge of whatever xml's exist (up to potentially 5) and copies to end result, as MS-DOS.xml, over to the data folder.

That would ensure that the user isn't involved beyond simply downloading the torrent to the right folder. How does that sound?

[NLS]

Option three could be 31 different pre-made-once XML.

[StammesOpfer]

Ok so my thought to make this super simple. Have files RPG.game Adv.game and RPG.app Adv.app. I am doing this without and example file right now but the logic is correct. Might need to tweak names and location.

 

Batch file

echo ^<games^> > MS-DOS.xml

type *.game >> MS-DOS.xml

echo ^</games^> >> MS-DOS.xml

echo ^<apps^> >> MS-DOS.xml

type *.apps >> MS-DOS.xml

echo ^</apps^> >> MS-DOS.xml

[eXo]

Yea, the logic here works. Simplifying it down to this seems to work great:

echo ^<?xml version="1.0" standalone="yes"?^> > MS-DOS.xml
echo ^<LaunchBox^> >> MS-DOS.xml
type *.game >> MS-DOS.xml
type *.app >> MS-DOS.xml
echo . >> MS-DOS.xml
echo ^</LaunchBox^> >> MS-DOS.xml

 

Thank you for your input!

Edited December 6, 2017 by eXo

[StammesOpfer]

Ah. Yeah that works. Couldn't remember how the XML was formatted off the top of my head and not at home.

[eXo]

I've just tested it with several different setups and various xml files and it works great. I'm really happy to see there was a modular solution to this.

With that, I believe the torrent is ready for release. When I get home today I'll copy all of the files over to my seedbox, hash it, and if I have time, post it. However the copying and hashing are long processes and it may not be until tomorrow.

In the mean time, here is the file structure and setup. I'm trying to anticipate how it all works when I have all 5 volumes, so typing it out and getting some eyes on it is probably a good thing.
 

The folder contains the following:

\eXoDOS\

Adventure Collection ReadMe.txt

AdvMetadata.zip

LaunchBox.zip

Setup.bat

 

The eXoDOS subfolder is essentially the old eXoDOS folder. Inside it has a .\Games\ and .\Util\ subfolder. Users with files from my previous collection should be able to place them here in order to not have to redownload from scratch.

The AdvMetadata.zip file are the parts of the LaunchBox frontend that appear to be dynamic to me.

The LaunchBox.zip file are the parts of the LaunchBox frontend that I believe will be static.

In other words, I believe the LaunchBox.zip file will be the same across all 5 volumes, while the AdvMetadata.zip file contains folders that contain data unique to this collection.

The metadata zip file has the following folders:

.\Data\

.\Images\

.\Manuals\

.\Metadata\

.\xml\

 

The next volume will then have an RPGMetadata.zip file, which will have these same folders in it. When is is extracted, it will add it's contents to these folders. 

The LaunchBox.zip file contains the following folders:

7-Zip, AutoHotkey, Backups (emptied), CDRDAO, DOSBox, Music, Plugins, ScummVM, Sounds, Themes, Updates, Videos, VLC

As none of these folders contain information about the games, they should be the same for each release. If I am incorrect in that assumption, please let me know.

 

Finally, there is the setup.bat file. It is a slightly modified version of my old setup file. Running this extracts the launchbox.zip file, any present metadata.zip files (it checks for all 5 volumes), and then creates a merged MS-DOS.xml file for any volumes present and copies this over to the .\Data\Platforms\ folder.

It then proceeds to unpack the necessary files for eXoDOS to run inside the .\eXoDOS\ folder. It also creates an ini for meagre to run for those folks who still want to access things the old way, or just compare data between the two front ends, or whatever. I also have several people who have made tools for the original meagre setup, so I have retained it.

 

Launchbox itself is treating each games launch.bat file as "the game", rather than the actual game archives. This allows the launch of a game to retain all of the conf settings I have previously created for dosbox as well as batch file options to choose between dosbox or scummvm for certain games, or which episode of a game to launch (for the multi-part games), or any other variations that I had to create for various games through the years. 

 

Hopefully this gives everyone a good idea of what to expect and how the setup works. Feedback is appreciated.