Jump to content
LaunchBox Community Forums

Recommended Posts

2 hours ago, oblivioncth said:

The change started with 10.12 beta 1, and right now it's on beta 8 I think? So stable 10.12 should be out soon if you dont want to download the beta.

Ok, to make the fixes that I want and maintain the portability, I will have to break the current way I'm doing doing things.  Probably good time to do that. This means people will have to redo their ScummVM setup with the updated plugin once its done.  I plan to move everything into the scummvm folder where ever it lives for 10.12.  Once you've imported your games via the bat files.  I plan to add an option to convert your batrunner imported games to launch scummvm straight from launchbox skipping the batrunner and bat files.  I believe the pause features will then work for those who want them while still maintaining portability with scummvm games and my scummvm configuration utility that converts the scummvm ini file to all relative paths.  Thoughts on this approach?

Share this post


Link to post
Share on other sites
4 hours ago, epicfail said:

Ok, to make the fixes that I want and maintain the portability, I will have to break the current way I'm doing doing things.  Probably good time to do that. This means people will have to redo their ScummVM setup with the updated plugin once its done.  I plan to move everything into the scummvm folder where ever it lives for 10.12.  Once you've imported your games via the bat files.  I plan to add an option to convert your batrunner imported games to launch scummvm straight from launchbox skipping the batrunner and bat files.  I believe the pause features will then work for those who want them while still maintaining portability with scummvm games and my scummvm configuration utility that converts the scummvm ini file to all relative paths.  Thoughts on this approach?

I wrote a MATLAB script to scan my scummvm ini and create .scummvm files with the short names and have just scanned in those. I'm content with doing this for the time being; however if I do end up switching back due to features in the future I think that approach sounds fine.

Share this post


Link to post
Share on other sites

Any progress on this?  Why not skip bat files entirely and just create the scummvm files that the emulators now use? In RA they don't have to use shortnames either, so creating them with long names would also let LB import them directly. Doing this instead of relying on folder name means the "Scan for Added ROMs" feature in LB can be used in the future - it can't if you scan folder names.

  • Like 1

Share this post


Link to post
Share on other sites

If possible, I agree with Pixelpiper, other frontends like ARC for android are using the .scummvm format as well since the LB android is dead in the water for now

Share this post


Link to post
Share on other sites
Posted (edited)

I've written a solution in PHP and I was hoping to make a Windows executable from it, but so far no luck.  The only php compiler I can get to work is stuck on PHP version 4 and the workarounds I have to do to make the code work fail with relative paths.

Here's the pseudo-code for a simple solution:

Read/parse ini into an associative array (or similar) - look in "Emulators/RetroArch/system/scummvm.ini" or if not found, then "%APPDATA%/ScummVM/scummvm.ini"

Foreach (loop) [sections], ignoring section "scummvm"
 	get values for description, gameid* and path
	sanitize invalid filename characters in description (replace slashes with dashes for example, get rid of colons, etc.)
 	make file (open file for writing) "path/description.scummvm"
 	write gameid into file
 	close file

* Libretro say to use the "gameid" (aka short name) - and it works. But, the short name isn't unique across different platforms/versions so it works because the current path is always the data file path for the game when using the .scummvm file. Using instead the longer version-specific name found in each ini "section" makes it possible to start from anywhere without specifying the path. 

Works beautifully in my case, but I had to test it on my unix server to make the files.  Using the description for the filename gets you ScummVM-derived unique names that include the language for the release. This is great when you have multiple versions of a game, without needing to rely on the folder names at all. From within LB you can scan all *.scummvm files and they'll be recognized with correct names and you should be able to launch any combined versions independently.

Edited by Pixelpiper

Share this post


Link to post
Share on other sites

I'm working on something...  Since I can't easily make the script portable right now, I'm going to make a set of the ".scummvm" files you can copy directly into your scummvm games top-level folder. These files are both scrapable by LaunchBox and directly launchable via RetroArch

The folder names will match those in downloads from Ghostware and Replayers. 

Examples:

"Beneath a Steel Sky (Floppy VGA)" and "Beneath a Steel Sky (CD VGA)"

and

"Broken Sword 1 The Shadow of the Templar (CD PS1)" and "Broken Sword 1 The Shadow of the Templar (CD Windows)"

I'll link a new  thread when done and tested.

Share this post


Link to post
Share on other sites
Posted (edited)
On 3/28/2020 at 11:19 PM, oblivioncth said:

I wrote a MATLAB script to scan my scummvm ini and create .scummvm files with the short names and have just scanned in those. I'm content with doing this for the time being; however if I do end up switching back due to features in the future I think that approach sounds fine.

Are you using actual short names (the first part of the section/target before "-") or are you using "gameid" ?    The gameid is, according to ScummVM dev docs, supposed to be unique, but they've seemingly ignored this and it doesn't specify the version at all as they indicated. In many cases it specifies an engine, like sci or agi and doesn't mention the game whatsoever.  The actual short names are in the game target (the [section] of the ini) - this is the only tag that will start a game on Stand-Alone ScummVM.  Libretro can start using either target or gameid.

Libretro docs clearly state to use the Short Name inside the .scummvm files - but as above, this is broken in the core and just won't work for some games.  For example, the short name for Bargon Attack is "bargon" which is shared in all versions (bargon for dos, bargon-amiga for Amiga, bargon-amiga-1 incorrectly for Atari ST, bargon-de for German DOS, etc.). If you try using that in a scummvm file you can start the DOS versions of the game, but Amiga and Atari ST version will drop out to the ScummVM debugger.  If instead you use the full target name or the gameid of "gon" which is identical for all versions, they'll all start correctly.

Using short name with Kings Quest 1, if you start the Amiga or Mac versions they'll each contain the correct copyright date, but they'll be using the darker DOS palette and DOS audio - start each game directly from the ScummVM launcher and you can see/hear the difference.

Kyrandia produces an error on Amiga and other versions, etc.  

Frankly, the whole thing is a giant mess and both ScummVM and the Libretro core are full of inconsistencies and bugs. I've now got a script going that deals with almost all special cases to properly tag a file for every game (I'm testing with over 400 game versions and over 250 titles) - this is only applicable to RetroArch/Libretro. All games that have any details in the LB/EmuMovies databases also download all metadata and artwork (almost everything except some non-game stuff).

WIth respect to stand-alone ScummVM, you can't start games by passing it an external file. But you can start it from an external batch file that can also be made with such a script - something as simple as "scummvm.exe <target_name>" - and naming that file appropriately will allow LB to scrape all details.

Edited by Pixelpiper

Share this post


Link to post
Share on other sites
Posted (edited)
5 hours ago, Pixelpiper said:

Are you using actual short names (the first part of the section/target before "-") or are you using "gameid" ?    The gameid is, according to ScummVM dev docs, supposed to be unique, but they've seemingly ignored this and it doesn't specify the version at all as they indicated. In many cases it specifies an engine, like sci or agi and doesn't mention the game whatsoever.  The actual short names are in the game target (the [section] of the ini) - this is the only tag that will start a game on Stand-Alone ScummVM.  Libretro can start using either target or gameid.

Libretro docs clearly state to use the Short Name inside the .scummvm files - but as above, this is broken in the core and just won't work for some games.  For example, the short name for Bargon Attack is "bargon" which is shared in all versions (bargon for dos, bargon-amiga for Amiga, bargon-amiga-1 incorrectly for Atari ST, bargon-de for German DOS, etc.). If you try using that in a scummvm file you can start the DOS versions of the game, but Amiga and Atari ST version will drop out to the ScummVM debugger.  If instead you use the full target name or the gameid of "gon" which is identical for all versions, they'll all start correctly.

Using short name with Kings Quest 1, if you start the Amiga or Mac versions they'll each contain the correct copyright date, but they'll be using the darker DOS palette and DOS audio - start each game directly from the ScummVM launcher and you can see/hear the difference.

Kyrandia produces an error on Amiga and other versions, etc.  

Frankly, the whole thing is a giant mess and both ScummVM and the Libretro core are full of inconsistencies and bugs. I've now got a script going that deals with almost all special cases to properly tag a file for every game (I'm testing with over 400 game versions and over 250 titles) - this is only applicable to RetroArch/Libretro. All games that have any details in the LB/EmuMovies databases also download all metadata and artwork (almost everything except some non-game stuff).

WIth respect to stand-alone ScummVM, you can't start games by passing it an external file. But you can start it from an external batch file that can also be made with such a script - something as simple as "scummvm.exe <target_name>" - and naming that file appropriately will allow LB to scrape all details.

Hmm, I personally haven't experienced any of these issues but I'd believe they exist.

I use standalone 2.1.1 so I can't speak for how Retroarch handles starting games, though I did try to mold my setup in a way that would make it easy to change to Retroarch's ScummVM core if I so choose. I don't have access to my setup at the moment but will later today and I don't want say anything about how I ultimately imported my ScummVM games that is untrue in the event that I remember something wrong. So, once I'm in front of it later tonight I'll check the couple details I'm fuzzy with and get back to you. I remember most of how I handled it, I just want to double check a few things.

I have a full set so I know what you mean when it comes to the duplicate short names, but at least in 2.1.1 that is a non-issue mostly.

Edited by oblivioncth
  • Like 1

Share this post


Link to post
Share on other sites
Posted (edited)
17 hours ago, oblivioncth said:

I use standalone 2.1.1 so I can't speak for how Retroarch handles starting games, though I did try to mold my setup in a way that would make it easy to change to Retroarch's ScummVM core if I so choose. I don't have access to my setup at the moment but will later today 

I have a full set so I know what you mean when it comes to the duplicate short names, but at least in 2.1.1 that is a non-issue mostly.

Thanks, take a look and let us know.  I have everything working well now to launch everything using Stand-Alone ScummVM (I've given up on using RetroArch for this personally) and just wrestling with splitting a few titles in LaunchBox and trying to make sure they come up as different re-releases (Space Quest 1 for example).

The stuff I mentioned previously should only be a factor for RetroArch.

There are other potential issues with using ScummVM stand-alone and a few ways to accomplish that - I decided to go the route of cmd files that do the launching.  I also tried another method, using filenames named with the Target names and then letting LaunchBox look at folders for Game Name - it works well and doesn't require any scripts, but will screw up later if you split any combined titles.

One thing I haven't been able to figure out is how to use LaunchBox's built-in ScummVM tab.  Never found anyone talking about it and just couldn't get i to work. Also found some other issues in LB which are annoyances and should be fixed, but all can be gotten around.

Edited by Pixelpiper

Share this post


Link to post
Share on other sites
7 hours ago, Pixelpiper said:

-snip-

Ok so here is the deal. Like you mentioned, every game that ScummVM officially supports has a (sort of) unique identifier that is its short name. While I don't remember what the ScummVM docs say about this, what I am referring to is the abbreviated name that is used as the header for each game entry within your scummvm.ini. There is a "gameid" for each entry, but this is most certainly not unique as it seems to me like it is more of an "engineid", i.e. telling the Scumm engine which base interpreter to use for that entry, and obviously there are only so many of these that are shared between various games within the ScummVM set.

To clarify, here is an example of the entry for the CD/DOS/German version of "Indiana Jones and the Fate of Atlantis" in my INI:

[atlantis]
description=Indiana Jones and the Fate of Atlantis (CD/DOS/German)
path=D:\LaunchBox\RAI\ScummVM\Standard\German\Indiana Jones and the Fate of Atlantis (CD DOS, German)\
gameid=atlantis
language=de
guioptions=lang_German

The shortname I am referring to is "atlantis" here. Funny enough a lot of these short names seem to be undocumented, or at least are hard to find.

Now, as much as this system seems to work like how MAME handles roms where each game has a truly unique shortname that you can just pass as an argument when starting it and as long as its in your ROMs directory the game will be started without needing to provided a direct path to the rom, it seems that these are rather just default or "preferred" names for each game. This is demonstrated in two ways: First, when you have multiple versions of a game ScummVM, while scanning them in, handles them just like most file systems do when something automatically gets assigned a name that is already taken; it simply appends a number onto the end, with the number simply corresponding to the order in which the versions were found during the scan.

Correspondingly, here is all of the entries of have for my versions of the above game: https://pastebin.com/RWK5UT4x

The "(CD/DOS/German)" version was found first due to its alphabetical position from its folder name, so it was assigned atlantis, while the "Floppy/DOS/English" version was found second during the scan (again due to its folder name) so it was assigned atlantis-1. You will notice that some versions also have their own short names, like atlantis-amiga, but ultimately these are handled the same way in that if you have multiple versions, of that version, -N (where N is the next available number) will simply be appended to the end. So unlike MAME where each game has a fixed internal name that is used to launch it, the short names in ScummVM are more or less an an index for each individual's scummvm.ini to let ScummVM which entry to refer to when starting. There tends to be a lot of overlap between any two person's INIs due to the "prefered" names that are used, but as you add variants of games to any INI, the exact short names that appear will differ depending on the exact variants and folder names that an individual has/used.

So this is how you handle launching any specific version of a game. You have to check what short name it received within your INI and then use that in the command line to start it.

Second, use of the "preferred"/default short names is not at all enforced. In my INI I can change atlantis to thisisadifferentname and then do:

scummvm.exe thisisadifferentname

and it will still launch the CD DOS German version of Indiana Jones and the Fate of Atlantis.

Again, this reinforces the idea that while there are default names, the short names ultimately used for each game are technically unique to each install and are not universal.

As for how I got my collection into Launchbox correctly, I did the following:

  1. Add standalone ScummVM 2.1.1 as an emulator with No Quotes and Name Without Extension checked
  2. Create a new empty folder to hold onto named whatever you want (in my case, a folder called "Launch Files" under the root of where my ScummVM game files are stored
  3. Create a file/folder hierarchy with sole purpose of adding the games to Launchbox in the most optimal way. This is done using my MATLAB script which is explained in the following steps
  4. Iterate over each game entry within the INI, which is easy since they always start with '[', and store the short name and data in the  "description=" field to their own variables.
  5. For each entry, create a folder under that new root folder (i.e. Launch Files) with the name from the description=" field, and then within that folder create a file with the entries short name and ".scummvm" as an extension
  6. Optionally, for better compatibility with Retroarch ScummVM, write the short name of the game only to that file (i.e. atlantis.scummvm will contain the text "atlantis" and nothing else)
  7. Once the script is finished, go into LB and import using the "ROM" files option, point it to the root folder used (Launch Files), tell it to scan sub folders, and tell it to use the folder names as the game title instead of the file names

This way, each game gets the correct title, and the "rom" file for each game in my LB database is effectively shortname.scummvm, which makes it clear what the file is for in my file-system, but since the "Don't use extensions" option is checked only the short name will be passed to scummvm.exe when starting a game. So while that whole folder/file structure you end up creating holds no actual game data, it is ideal for getting games to start correctly with launchbox and ensuring each game receives the correct metadata. This has worked perfectly for all my ScummVM games. 

Now, I did need to do some manual auditing of my game folder names and the "description=" fields in my INI (these are also not strict and you can change them to whatever you wish) due to some games that used the same exact description (I used a separate script to find these) and clean up a few other things, but once that was done the main script took care of 95% of the work. If it helps clarify what I did you can see it here:

https://pastebin.com/y4tMrh0j

Share this post


Link to post
Share on other sites

So we're exactly on the same page.  I did precisely what you described during my testing, which I'd mentioned in the post from earlier today.  Folder-derived names unfortunately fall apart if you have to split combined games.  The split games will end up with the names of the files, so "Something" and "Something-amiga" for example, and lose the folder-derived name.

Those header are referred to as Targets.  They're made of the short name plus extra text.  This isn't what the RetroArch/Libretro docs indicate as I believe they're conflating it with the gameid.  ScummVM docs describe the purpose of gameid as a unique identifier, which we both know it's not.  Nowhere in those docs do they mention it's ever supposed to include anything about the engine used.

In my script I've accounted for some 99% of special cases so that most things don't have to be tweaked in LaunchBox and it's now making files for stand-alone ScummVM and RetroArch, both inside game folders and/or outside (when I think about it, it's not realy cool to be putting files in the folder that essentially makes the "ROM"). It already changes some game names based on originally crap descriptions, etc.  I might consider converting it to Auto Hot Key from PHP but I'm waiting on some info right now from a company that has a commercial PHP compiler available for Windows.

 

 

  • Like 1

Share this post


Link to post
Share on other sites

Ah sorry, since there was a bit of ambiguity as to what you were referring to given the poorly defined ScummVM terms due to the lack of documentation, I figured I'd just explain everything I did to be safe. Wasn't aware the entries were referred to as Targets. Good to know. 

I wasn't aware of that issue that LB would have while splitting games. Since each game ultimately has its own ID there would be no issue with keeping the folder derived name on both. I guess after splitting there will be some manual adjustments needed anyway, during which you can fix the name, but that is definitely still annoying.

I agree with you in that I think whoever made the Retroarch docs for ScummVM is conflating gameid and short names. I am also surprised about the lack of documentation concerning many games that ScummVM definitely supports.

It is a shame that so much processing is require to get the games in correctly but I suppose it's still better than having to add them all one-by-one.

Share this post


Link to post
Share on other sites

Your steps @oblivioncth captured very well what I eventually broke down and did after the new ScummVM version came out.  Once you resort to normal EXE as emulator and pointer text files, then get benefit of Pause feature in LB as well as "future proof" EXE wise.  The few new games they add over time are easy enough to add. If I was coder I am sure it would have been easier, but having no skill in that area took the brute force approach of doing each folder and modifying the text file in each to match the ScummVM name.  Not elegant but worked and don't have to mess with it anymore. The pointer files usable with standalone ScummVM and Retroarch and folder structure are in ZIP file I attached in this thread.  

Share this post


Link to post
Share on other sites

So here's some additional information I hadn't mentioned before, but oblivioncth alluded to above...  There's no guarantee that the Target names will be exactly the same between a RetroArch ScummVM core install and the standalone as far as I can tell - that can also depend on how  you add everything in and what you're adding.  One way to make sure everything is stable is to make the ini with one or the other and then copy it over to the one that wasn't used.

Further to this, I don't even keep the ini in the default APPDATA location for standalone, I force the use of one inside the folder that also holds the launcher files. I did this to try and keep the whole thing more portable.  Even though the ini would have to be adjusted because all the game data locations are included with absolute paths.

The big benefit in having a script or app to do this work is in the case where you have to remake the ini for any reason. It would suck to remake it and have the Target names change so that they no longer work with any launcher files and then having to redo it all manually - again.

Am I correct in thinking the only way to add executable (or batch) files to LaunchBox is by dragging them in? I don't see anything else that works the same way from a menu - none of the import options for example will do this.  If using Import ROM on batch files, you have to go back afterwards and turn off "Emulator" for every file (which can be done in batch, but still an extra step).

Share this post


Link to post
Share on other sites

It'd be "REAL NICE" if maybe Jason could update the built-in ScummVM support so it's open-ended.  Remove all the hard-coded Target info from Launchbox and have its list of Targets read directly from the ScummVM ini - which you should be able to point LB to.  The ini has to exist already anyway to use any of these games in LB, otherwise ScummVM itself wouldn't start the games, AFAIK.

That said however, I still don't even know how to import a batch of games all at once for the built-in ScummVM support.  If it has to be done one game at a time, I guess it doesn't really matter, it wouldn't be useful anyway.

Share this post


Link to post
Share on other sites
8 hours ago, sundogak said:

Your steps @oblivioncth captured very well what I eventually broke down and did after the new ScummVM version came out.  Once you resort to normal EXE as emulator and pointer text files, then get benefit of Pause feature in LB as well as "future proof" EXE wise.  The few new games they add over time are easy enough to add. If I was coder I am sure it would have been easier, but having no skill in that area took the brute force approach of doing each folder and modifying the text file in each to match the ScummVM name.  Not elegant but worked and don't have to mess with it anymore. The pointer files usable with standalone ScummVM and Retroarch and folder structure are in ZIP file I attached in this thread.  

Definitely convenient for people who are starting fresh, and yea this seems like the way to go.

8 hours ago, Pixelpiper said:

It'd be "REAL NICE" if maybe Jason could update the built-in ScummVM support so it's open-ended.  Remove all the hard-coded Target info from Launchbox and have its list of Targets read directly from the ScummVM ini - which you should be able to point LB to.  The ini has to exist already anyway to use any of these games in LB, otherwise ScummVM itself wouldn't start the games, AFAIK.

That said however, I still don't even know how to import a batch of games all at once for the built-in ScummVM support.  If it has to be done one game at a time, I guess it doesn't really matter, it wouldn't be useful anyway.

I don't know of any other way to import such things without drag and drop other. I don't really know how exactly the built-in version handles things but I had no idea that its targets were hardcoded. This is definitely an issue since like we said the names are not at all guaranteed to be the same between systems.

No clue if Jason will ever have the time or care to get to it, but a built-in import option that reads the INI directly and more or less does what we did through scripting would ultimately be ideal.

Share this post


Link to post
Share on other sites
5 hours ago, oblivioncth said:

No clue if Jason will ever have the time or care to get to it, but a built-in import option that reads the INI directly and more or less does what we did through scripting would ultimately be ideal.

If the built-in system was enhanced, then the launching could definitely work in a similar way to what we've done, but with the benefit of not needing any external bat/cmd/launch files - all the target names could be stored within LaunchBox's settings files/databases. Likely not so different than the built-in support works now, except no need to hard-code a list of games.

I'm adding the games that were made compatible in the last ScummVM release to my collection right now, so seeing first hand how well my process works for "updating" an existing platform. I don't expect any issues, but we'll see. :)

  • Game On 1

Share this post


Link to post
Share on other sites
Posted (edited)

I've made a script to be used instead of an exe in LaunchBox as a ScummVM emulator. The script will take any command-line parameters and pass them to the ScummVM exe. It will also take a ".scummvm" file (pointer file) we've been talking about and extract the Target ID from inside it and pass that to ScummVM as well. Lastly, it tells ScumVM to maintain its settings/ini in its current directory. In addition to these benefits, using this script produces no command prompts/windows - so you get a clean full screen ScummVM launch and can still use all the normal LaunchBox features/options like for other emulated platforms.

Another script can be used to open ScummVM from Explorer and specifies the same local settings/ini.  An archive with these scripts and pointer files for a near-complete full set can be found in the following thread along with instructions: 

 

IMO, this is the easiest and cleanest way to handle ScummVM right now.  The only thing missing is that I can't upload my script to make the .scummvm files right now - it's still in PHP and I need to get it into a format anyone can easily use.

Edited by Pixelpiper

Share this post


Link to post
Share on other sites

Thanks for this discussion, folks.

I just tried to load one of my ScummVM games and found the system was now broken with the current LB releases. It had been a while since I played one so I had not noticed it was a broken mess. I was using the plugin by epicFail which was a huge time saver and help to the community.

I just wanted to comment on your current discussion and let you know that there are people out here happy that you are discussing the issues and we hope that this can get cleaned up and working for us normal mortals :)

Thank you.

-David

Share this post


Link to post
Share on other sites
On 6/22/2020 at 12:58 AM, drswoboda said:

Thanks for this discussion, folks.

I just tried to load one of my ScummVM games and found the system was now broken with the current LB releases. It had been a while since I played one so I had not noticed it was a broken mess. I was using the plugin by epicFail which was a huge time saver and help to the community.

I just wanted to comment on your current discussion and let you know that there are people out here happy that you are discussing the issues and we hope that this can get cleaned up and working for us normal mortals :)

Thank you.

-David

Thanks for the kind words.  Thank you for being so patient with me, I will get the update done.  I'm very mortal, but some will argue the normal part.  Sometimes you just need to drink too much beer, enjoy your family, and ignore the world. :)  

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...