Nielk1

@Klopjero It currently works on the assumption that the files are all supposed to be zips from the Doomworld database. For this reason, it takes the input file and changes the extension to zip (I've not limited this yet to only do this to 7z files, and I've not yet considered searching for WADs) then sends that zip filename to Doomworld. Doomworld will get a little broad though, for example "foobar.zip" will match a search for "bar.zip". If the plugin finds that only 1 result from Doomworld has an exact filename match, it will use it. If the plugin finds multiple matches, even partial ones, it will store it for after the perfect matches, then ask. Only if there is no result at all will it be skipped.

I also noticed something odd. If you use Ctrl+A to select all the games you want to scrape with it LaunchBox doesn't update the SelectedGames in the API helper classes. To get the titles to update I have to select all the games and press F5. I will see if there is an update function I can call.

Thanks Cid. If I took a screenshot it would be in a different place due to my other plugins that move stuff around.

If you use discord feel free to drop by the Launchbox discord to hit me up there or to DM me. I might have to recompile the DLL or something for a differnt version, or I might have to do something else. I'm hoping the next LaunchBox release version is soon so I can stop futzing around with "maybe this is why it isn't showing" and know for sure if my code is running awry. Do make sure you're putting it in LaunchBox's Plugins folder and not in a Theme's Plugin folder.

It should be the last item in the Tools menu then. It's not there? It says Doomworld Scraper and has the site's icon as its menu icon. Never had this issue before, ever. How very odd. No frustration, just bemusement. I wonder if I screwed something up.

It doesn't normally, only if the menu cleanup plugin is installed.

Folder structure's irrelevant, Launchbox aggressively searches for all DLL files in its Plugins folder. The menu item will either be on the bottom of the [Tools] menu, or under [Tools > Download Metadata and Media...] (which will now have child items) if you have the menu cleanup plugin as well. I am working with the LB beta so it's possible some difference is preventing it from working.

@Klopjero All you need to do is place the DLLs from bin into a subfolder in Plugins. Launchbox will see them when it starts and add the menu item automatically. The source code is there for if I disappear or something or if for some reason you don't trust me and want to compile it yourself. I have an extra step in my system that copies the DLLs from the project's bin into a folder in Launchbox/Plugins before starting Launchbox in my debugger.

@Klopjero If you want to test it now you can. There's probably a huge pile of bugs in it at this stage. This zip contains the source code, in the bin folder are the plugin DLLs you will need to add to LaunchBox's Plugin folder. You can make a subfolder for this in there. You should consider this very-beta and as with any beta things you should back up your Data folder. LaunchBox_DoomWorldScraper.zip

So I've been trying to avoid using JSON.net just due to not wanting to conflict with any other plugins that use it as during the LaunchBox current beta I am not sure what version to target. Welp, JSON.net is just too good. I switched to that from the native JSON parser and was able to fix my issue immediately by writing a custom serilalizer for attributes that could be either arrays or singles. I hope the next version of LaunchBox releases soon so I can fix my version targets. I'm done for tonight but here's a picture. I will probably want to do some followup API calls using that Id to get what's not listed here. What suggestions do you have for what to do with the data? You can see the types of data by looking at various items there such as this one: https://www.doomworld.com/idgames/levels/doom/Ports/0-9/007ltsd I'll probably allow for everything to be added as custom fields so complex themes can access it of course, or more specifically I will add some sort of check-boxes to the scraper for it.

No, just good coding practice.

I hit a snag with the JSON data format. I am going to try to use the XML instead. The problem is that some items, like files lists are an Object if it's a single item or an Array of Objects if it's more than one. The result is all my checks are seeing 0 results because it's failing to read the response. The good thing is that the way I coded the API it's probably only 2 small sections that need editing. probably.

Things are going well. Now I am just trying to figure out what to do with the matches as they occur. I'm thinking I could do it like a windows copy dialog works, where all perfect matches will go through first, then it will start asking questions.

I've gotten this far, with the dialog poking the API to ask if it's online and get its information. I of course still need to simulate certain errors and such but probably won't get to that till it's working. And yes, I see it should be Doomworld not DoomWorld.

So, I'm still working on this. For some reason I decided to wrap the entire API up in a nice interface. It works, so now I just need to stick everything together.

O ye, of little faith. I'll try to work on this tomorrow (later today) after work. As noted it looks pretty simple.

The plan is to scrape everything that can reasonably fit into LaunchBox, possibly even utilizing custom fields for those with Premium. If there's a lot of ambiguity I can give the user more control of the process, if there's less I can make it almost entirely automatic. I won't know exactly how I am inserting the data into LaunchBox until after I have it. My goal is to first make the scraper collect the data for the selected games, then worry about how to actually integrate that data.

C# is a lot easier than C++.

@Klopjero LaunchBox/BigBox is written in .net, so the plugins are as well (except for a complicated scenario that hasn't occurred). I use C# as IMHO it's the best .net language. (C# looks very similar to Java.)

@Klopjero I recently learned about the 4:3 vs 16:9 thing with hyperspin while trying to write a plugin that can use said themes (or something of similar functionality) in BigBox. Is there a 4:3 version the assets can be extracted from? That would avoid the loss of quality from stretching the images back out.. My plan is to get the scraper plugin working at the level needed, then return to my more complex projects with the new data in hopes of creating a cooler future plugin. I'd of course return to this plugin to fix any issues as this is way simpler than my other projects. This plugin will have its source code on GitHub.

@Klopjero I've loaded enough data (very minimal, just the "roms" and clear logos) into my LaunchBox to use as a working base and I have tested the API with success. All that remains is the actual implementation. If I can trust the zip/7zip to have a good filename I can implement the needed logic quite reasonably. For example, here is a call using one of the filenames. I could also do a far slower search if needed where I look inside the 7z/zip files. https://www.doomworld.com/idgames/api/api.php?action=search&query=007LTSD&type=filename&sort=date https://www.doomworld.com/idgames/api/api.php?action=search&query=007LTSD&type=filename&sort=date&out=json While I might have larger plans in the future (I am overzealous when it comes to such plans) I figure I can have a basic scraper working and released relatively quickly when I find the time to sit down and work on it. I can probably get it working in just a few hours of work.

I'm trying to build cross-ID maps between various databases as well as collect data that's not often used for custom fields to be displayed in BigBox theme elements. Showing the award icons on these items would be interesting. I need to make a hyperspin forum account so I can look at what you've linked. I've not used non-stock Doom WADs in a long time so I need to relearn a few things.

I'm interested but busy. I'll put this on my list if no one else does anything. At the very least I will be adding that API/Database to my little 'unified database project' that I have posted absolutely nothing about

@massatomic I made a plugin that, if another plugin is set up for it, will clean up their place in the menu bar. I figure if anyone has a good feature request I can throw that in there too. It's really simple right now and technically other plugins could do this work themselves but I figured a passive API to do it would be better.