[request] idGames Archive Public API

[Klopjero]

Hi, 

I'm working on porting the The Ultimate Single player Doom experience to Launchbox. This project, originally a labour of love by Zero Jay and later worked on by me, is the best of single player Wads that Doom Community has to offer. The Doom Community have held an annual event where they declare the best wads of that year. It's called the cacowards. https://www.doomworld.com/cacowards/ . and it's absolutely amazing. 

The set was made for Hyperspin, so it has a Hyperspin database, which you can't use in Launchbox sadly. 

but, Id has an Archive API, and most of these wads are stored by Id. 
https://www.doomworld.com/idgames/api/

Now, my not so modest question, Could a very talented coder with spare time, perhaps build a scraper that can feed the Launchbox database ?

Edited September 1, 2017 by Klopjero

[Klopjero]

Such interest.

"wow klop, tell us more about this amazing collection"
Allright I will, This Collection is a tribute to the amazing work that has been put out by DooM community over a time period of 27 years! Isn't it amazing ? what other games have such a loyal following that it's still being actively developed after 26 years!
Seriously, that's amazing. This Collection is the best of best. The cacowards is a community prize that is given to the creators of the best wads released that year.

I'm very excited because it's nearing december and the winners will be announced!
For LB I'm working on gathering resources like screenshots and clear logo's, banners and custom plates for the "platform" and "collection". If only I possessed the Programming skils to make use of the community scraper, that would enrich the database vastly, tapping directly into the Doomworld archive. 

The plates I'm making are based on @nosh's NoshonThis collection, it looks so cool! I'm so psyched! Rip'n Tear baby!
 

[Nielk1]

I'm interested but busy.

I'll put this on my list if no one else does anything.  At the very least I will be adding that API/Database to my little 'unified database project' that I have posted absolutely nothing about

[Klopjero]

7 hours ago, Nielk1 said:

I'm interested but busy.

I'll put this on my list if no one else does anything.  At the very least I will be adding that API/Database to my little 'unified database project' that I have posted absolutely nothing about

please enlighten me.

[Nielk1]

I'm trying to build cross-ID maps between various databases as well as collect data that's not often used for custom fields to be displayed in BigBox theme elements.  Showing the award icons on these items would be interesting.

I need to make a hyperspin forum account so I can look at what you've linked.  I've not used non-stock Doom WADs in a long time so I need to relearn a few things.

[Klopjero]

23 minutes ago, Nielk1 said:

I'm trying to build cross-ID maps between various databases as well as collect data that's not often used for custom fields to be displayed in BigBox theme elements.  Showing the award icons on these items would be interesting.

I need to make a hyperspin forum account so I can look at what you've linked.  I've not used non-stock Doom WADs in a long time so I need to relearn a few things.

here https://zerojay.com/nextcloud/index.php/s/zbGHCHwwS07JJ81

[Nielk1]

@Klopjero

I've loaded enough data (very minimal, just the "roms" and clear logos) into my LaunchBox to use as a working base and I have tested the API with success.

All that remains is the actual implementation.  If I can trust the zip/7zip to have a good filename I can implement the needed logic quite reasonably.  For example, here is a call using one of the filenames.  I could also do a far slower search if needed where I look inside the 7z/zip files.

https://www.doomworld.com/idgames/api/api.php?action=search&query=007LTSD&type=filename&sort=date
https://www.doomworld.com/idgames/api/api.php?action=search&query=007LTSD&type=filename&sort=date&out=json

While I might have larger plans in the future (I am overzealous when it comes to such plans) I figure I can have a basic scraper working and released relatively quickly when I find the time to sit down and work on it.  I can probably get it working in just a few hours of work.

Edited September 7, 2017 by Nielk1

[Klopjero]

not all r

1 minute ago, Nielk1 said:

@Klopjero

I've loaded enough data (very minimal, just the "roms" and clear logos) into my LaunchBox to use as a working base and I have tested the API with success.

All that remains is the actual implementation.  If I can trust the zip/7zip to have a good filename I can implement the needed logic quite reasonably.  For example, here is a call using one of the filenames.  I could also do a far slower search if needed where I look inside the 7z/zip files.

https://www.doomworld.com/idgames/api/api.php?action=search&query=007LTSD&type=filename&sort=date
https://www.doomworld.com/idgames/api/api.php?action=search&query=007LTSD&type=filename&sort=date&out=json

While I might have larger plans in the future (I am overzealous when it comes to such plans) I figure I can have a basic scraper working and released relatively quickly when I find the time to sit down and work on it.  I can probably get it working in just a few hours of work.

That's great! most of the WADs are zipped using the same filename. this was done for accurate and easy finding of the wad on doomworld. however the gameplay mods, like brutal doom, brutal doom 64 or doom 4 doom, are not stored on doomworld. but have their own moddb entry. so those will have to be added manually. still there are exceptions when I ran a scrape through wadmanager not all wads were found. maybe they are ill named or maybe the doomworld api returned a denial of service because of the rapid querying.

This is set IS awesome, I do recommend it be used through Rocketlauncher. as it will save a buttload of configuring as all wads are already stored in it's ini's.

One thing that's on my fix list is updating the clearlogo's so that they are displayed properly. You have to understand that this is a hyperspin set, and Hyperspin has no native widescreen support, so to work around that, artwork like these clearlogo's are resized to 75% of it's original width to compensate for the stretching of the 4:3 canvas to a 16:9 aspect ratio.

[Nielk1]

@Klopjero I recently learned about the 4:3 vs 16:9 thing with hyperspin while trying to write a plugin that can use said themes (or something of similar functionality) in BigBox.  Is there a 4:3 version the assets can be extracted from?  That would avoid the loss of quality from stretching the images back out..

My plan is to get the scraper plugin working at the level needed, then return to my more complex projects with the new data in hopes of creating a cooler future plugin.  I'd of course return to this plugin to fix any issues as this is way simpler than my other projects.  This plugin will have its source code on GitHub.

[Klopjero]

I

5 minutes ago, Nielk1 said:

@Klopjero I recently learned about the 4:3 vs 16:9 thing with hyperspin while trying to write a plugin that can use said themes (or something of similar functionality) in BigBox.  Is there a 4:3 version the assets can be extracted from?  That would avoid the loss of quality from stretching the images back out..

My plan is to get the scraper plugin working at the level needed, then return to my more complex projects with the new data in hopes of creating a cooler future plugin.  I'd of course return to this plugin to fix any issues as this is way simpler than my other projects.  This plugin will have its source code on GitHub.

ah awesome , be sure to provide a link, what are you writing it in, in case I want to fork it.

I think I might still have the originals on a mega somewhere I'll have a to check.

[Nielk1]

@Klopjero LaunchBox/BigBox is written in .net, so the plugins are as well (except for a complicated scenario that hasn't occurred).  I use C# as IMHO it's the best .net language.  (C# looks very similar to Java.)

Edited September 7, 2017 by Nielk1

[Klopjero]

44 minutes ago, Nielk1 said:

@Klopjero LaunchBox/BigBox is written in .net, so the plugins are as well (except for a complicated scenario that hasn't occurred).  I use C# as IMHO it's the best .net language.  (C# looks very similar to Java.)

I'm in process of learning to code.. picked up c++, I think C# is similar to some degree.

Here are the 4:3's

Wheel.7z

[Nielk1]

C# is a lot easier than C++.

[Klopjero]

so I should definitely start with c++

So earlier you mentioned that you wanted to get the scraper working at the level needed. what info are you going to let it scrape?

title, ?
developer ?
description -> note ?
 

[Nielk1]

The plan is to scrape everything that can reasonably fit into LaunchBox, possibly even utilizing custom fields for those with Premium.  If there's a lot of ambiguity I can give the user more control of the process, if there's less I can make it almost entirely automatic.  I won't know exactly how I am inserting the data into LaunchBox until after I have it.  My goal is to first make the scraper collect the data for the selected games, then worry about how to actually integrate that data.

[Klopjero]

6 minutes ago, Nielk1 said:

The plan is to scrape everything that can reasonably fit into LaunchBox, possibly even utilizing custom fields for those with Premium.  If there's a lot of ambiguity I can give the user more control of the process, if there's less I can make it almost entirely automatic.  I won't know exactly how I am inserting the data into LaunchBox until after I have it.  My goal is to first make the scraper collect the data for the selected games, then worry about how to actually integrate that data.

not to sound demanding or know-it-all-y but, I've seen a different doomworld scraper that had a rather horrible error handling. when it was bulk scraping it dropped the batch when encountering an error.

If I may be so bold as to ask for feature like logging and resume on error.? in my head it would log the filename that it couldn't find with the error message.

[Nielk1]

O ye, of little faith.

I'll try to work on this tomorrow (later today) after work.  As noted it looks pretty simple.

[Klopjero]

I'm looking forward to your next post. on the side, you mentioned a hyperspin theme interpretor. that would be a nice addition.

do you know emubox? it's an opensource hyperspin-esq front end.. I'm not sure if it has been abandoned but the remnants of it can be found here.

https://github.com/LukeMcDonnell/EmuBoxFE/releases

[Nielk1]

So, I'm still working on this.  For some reason I decided to wrap the entire API up in a nice interface.  It works, so now I just need to stick everything together.

[Nielk1]

I've gotten this far, with the dialog poking the API to ask if it's online and get its information.  I of course still need to simulate certain errors and such but probably won't get to that till it's working.  And yes, I see it should be Doomworld not DoomWorld.

Edited September 8, 2017 by Nielk1