[request] idGames Archive Public API

[Nielk1]

It should be the last item in the Tools menu then. It's not there?  It says Doomworld Scraper and has the site's icon as its menu icon.  Never had this issue before, ever.  How very odd.

No frustration, just bemusement.  I wonder if I screwed something up.

Edited September 11, 2017 by Nielk1

[Klopjero]

I'll retrace my steps, and report bacK this evening.

[Nielk1]

If you use discord feel free to drop by the Launchbox discord to hit me up there or to DM me.  I might have to recompile the DLL or something for a differnt version, or I might have to do something else.  I'm hoping the next LaunchBox release version is soon so I can stop futzing around with "maybe this is why it isn't showing" and know for sure if my code is running awry.

Do make sure you're putting it in LaunchBox's Plugins folder and not in a Theme's Plugin folder.

[Klopjero]

6 hours ago, Nielk1 said:

If you use discord feel free to drop by the Launchbox discord to hit me up there or to DM me.  I might have to recompile the DLL or something for a differnt version, or I might have to do something else.  I'm hoping the next LaunchBox release version is soon so I can stop futzing around with "maybe this is why it isn't showing" and know for sure if my code is running awry.

Do make sure you're putting it in LaunchBox's Plugins folder and not in a Theme's Plugin folder.

[CriticalCid]

I just tested it and it shows up perfectly fine in my LB test installation running with the latest 7.11 beta version.

10 hours ago, Klopjero said:

Show me 2 pictures.

One of the folder structure of your plug in folder. The other of the context menu and where to find it so I can tell whether the plug in loaded properly. 

Thanks a lot so far.

[Nielk1]

Thanks Cid.  If I took a screenshot it would be in a different place due to my other plugins that move stuff around.

[Klopjero]

alot better

[Klopjero]

Okay the plug-in loads, hits are reported back... queery with multiple hits are returned as questsions.. top notch.. 

I noticed teh following: 

The <title> is set but it does not update in Launchbox. So 1994TU stays 1994TU until I edit and OK it ... then it displays the title properly 1994 Tune-up Community Project
Minor thing: Scraper doesn't say when it's done... it just stick at a full bar... as a user you expect it say it's done.
 

Edited September 11, 2017 by Klopjero

[Klopjero]

Apart from those things it works great! Nielk your awesome! 

[Nielk1]

I also noticed something odd.  If you use Ctrl+A to select all the games you want to scrape with it LaunchBox doesn't update the SelectedGames in the API helper classes.  To get the titles to update I have to select all the games and press F5.  I will see if there is an update function I can call.

[Klopjero]

Nielk, on what basis does the scraper decide to skip a file ? 

[Nielk1]

@Klopjero It currently works on the assumption that the files are all supposed to be zips from the Doomworld database.  For this reason, it takes the input file and changes the extension to zip (I've not limited this yet to only do this to 7z files, and I've not yet considered searching for WADs) then sends that zip filename to Doomworld.  Doomworld will get a little broad though, for example "foobar.zip" will match a search for "bar.zip".

If the plugin finds that only 1 result from Doomworld has an exact filename match, it will use it.
If the plugin finds multiple matches, even partial ones, it will store it for after the perfect matches, then ask.
Only if there is no result at all will it be skipped.

[Klopjero]

okay thanks... I will check my files. They might be named incorrectly then. 

[Nielk1]

If you have any suggestions that might improve how it works let me know.  I figure what I ended up with was probably the best method, just purely matching the zip names.  I am also forcing the names to lowercase as many of them were uppercase and they were lowercase on doomworld and thus not matching.  This could cause an issue if any of the ones in the Doomworld database are uppercase.

[Klopjero]

My files are named correctly. 

this is the behaviour. 

This is a search result that is succeful

and the 50shades one .. is skipped.
And this is the link to doomworld where the rom is stored : 
https://www.doomworld.com/idgames/levels/doom2/Ports/megawads/50shades

I don't understand. The skipped file doesnt show it's extension, is that the way it's supposed to be ? 

[Nielk1]

Oddly that one works fine for me, so here's my thoughts.

I have it skipping in the following situations:

1. No Emulator attached or ApplicationPath (path to ROM file) is not set.
2. If somehow making the filename end in .zip breaks it
3. The search call has no "Content".

If none of these occur it would write `Found {countFound} possible values`.  For this reason I suspect that one of these three is occurring.  It can't be #3 because it would say `Searching for {possibleZipName} ("{Title}")` and #2 is unlikely so it's probably #1.

If it's none of those I can try to improve the logging and fix some of the other issues, then provide another test version.  I do need to rewrite the concurrent logic and figure out why the progress bar never fills past 99% if it asks questions.

Once I've collected enough issues I will perform the refactoring.  I will also try to get this on github soonish as it's functional though clearly not finished.

[Klopjero]

Ah ha! then I think I've found the problem.

I had scanned to rom folder for new additions, because I was missing some WADs. (bloody LB doesn't remember where it had imported the roms from ( had to edit the path) ) and I suspect that the new wads don't have an emulator assigned to them. I will check this evening.

[Klopjero]

2 hours ago, Nielk1 said:

Oddly that one works fine for me, so here's my thoughts.

I have it skipping in the following situations:

1. No Emulator attached or ApplicationPath (path to ROM file) is not set.
2. If somehow making the filename end in .zip breaks it
3. The search call has no "Content".

If none of these occur it would write `Found {countFound} possible values`.  For this reason I suspect that one of these three is occurring.  It can't be #3 because it would say `Searching for {possibleZipName} ("{Title}")` and #2 is unlikely so it's probably #1.

If it's none of those I can try to improve the logging and fix some of the other issues, then provide another test version.  I do need to rewrite the concurrent logic and figure out why the progress bar never fills past 99% if it asks questions.

Once I've collected enough issues I will perform the refactoring.  I will also try to get this on github soonish as it's functional though clearly not finished.

Regarding logging, it might be helpful to give some hints so the user can check if he can fix the problem.

In this case, file is skipped, apparently you build in a check for an associated emulator, if the emulator is not found it could report that back.

[Klopjero]

it was the emulator.. this is a great niche scraper for launchbox! 

[Alerum68]

Hello!

Are you still working on this plugin at all?