srxz Posted September 27, 2018 Share Posted September 27, 2018 Steam Scraper View File Plugin that scrapes Steam Information, banner, screenshots and video to a selected game. Requirements: Launchbox 8.8+ Install: Extract the content of zip file to : Launchbox/plugins/SteamScraper Source code: Github (feel free to make a pull request ) Thanks to: @fawkesyeah for the extensive testing @CriticalCid for Ideas. EDIT If you have trouble running the plugin and crashing due CAS error, please right click on the DLL's , properties and then unblock. Usage: Just paste the Steam URL from the game that you need to scrape information. Video Tutorial: If you want to add Additional Custom fields and Tags please edit the properties.json : Note: This will generate a LOT of tags, I advise to hide it if you are going to use it. (Legacy Settings -> Game Details -> Disable Custom Fields) Also if you have the SteamDB or SteamSpy Link only you can use it to get the Steam metadata as well, the regex works just remind to put the / on the final url just like this: BUT THIS WONT GET INFO FROM THE RESPECTIVELY WEBSITES, IT WILL GO TO STEAM ANYWAYS. Submitter srxz Submitted 09/27/2018 Category Third-party Apps and Plugins 3 Quote Link to comment Share on other sites More sharing options...
Klopjero Posted September 28, 2018 Share Posted September 28, 2018 this works pretty good are you planning on building it out ? perhaps having the program scan the user profile for the games in the library, compiling a list and scraping off that list ? as an added function I mean. the individual scrape is pretty awesome as it allows for non steam-purchased games that are available on steam to be scraped with steam information. top notch! 1 Quote Link to comment Share on other sites More sharing options...
srxz Posted September 28, 2018 Author Share Posted September 28, 2018 4 hours ago, Klopjero said: this works pretty good are you planning on building it out ? perhaps having the program scan the user profile for the games in the library, compiling a list and scraping off that list ? as an added function I mean. the individual scrape is pretty awesome as it allows for non steam-purchased games that are available on steam to be scraped with steam information. top notch! Launchbox already has this feature to import every game from your user from steam, this scraper was made to update selected games, for example I have over 600 games on steam and only want 10~15 added to launchbox so I will add them manually and scrape it manually. But you got the point, it allows scrape steam games bought in other ways as well ? Quote Link to comment Share on other sites More sharing options...
Scriven Posted October 18, 2018 Share Posted October 18, 2018 I love this plugin, one of the few I use on a daily basis. Would it be possible for the plugin to check if the game's SteamID is present as the file name? The Astonishing Game has the ID 583190. The plugin could just add store.steampowered.com/app/ and the ID to automate the process. If the game wasn't imported by the Steam import through LB, it then could just ask for the URL as it does now. I have 6500+ games I have to update manually because most games are not in the LB database while the data is right there on the Steam website and while your plugin is awesome, I think it could be even better if you added this feature. Quote Link to comment Share on other sites More sharing options...
srxz Posted October 18, 2018 Author Share Posted October 18, 2018 3 hours ago, Scriven said: I love this plugin, one of the few I use on a daily basis. Would it be possible for the plugin to check if the game's SteamID is present as the file name? The Astonishing Game has the ID 583190. The plugin could just add store.steampowered.com/app/ and the ID to automate the process. If the game wasn't imported by the Steam import through LB, it then could just ask for the URL as it does now. I have 6500+ games I have to update manually because most games are not in the LB database while the data is right there on the Steam website and while your plugin is awesome, I think it could be even better if you added this feature. Hi Scriven, This is a good idea, since I never used the steam importer from LB I have no idea if adds the appID in somewhere, if yes I can do it easily. I will take a look this weekend on this. I didn't designed this for a 6500+ steam library, this is insane! ? Quote Link to comment Share on other sites More sharing options...
Scriven Posted October 18, 2018 Share Posted October 18, 2018 1 minute ago, srxz said: I didn't designed this for a 6500+ steam library, this is insane! ? Thanks, would be awesome if you could take a look. I know it's insane. You should see my backlog of games I still have to play someday. ? Quote Link to comment Share on other sites More sharing options...
srxz Posted October 18, 2018 Author Share Posted October 18, 2018 (edited) 46 minutes ago, Scriven said: Thanks, would be awesome if you could take a look. I know it's insane. You should see my backlog of games I still have to play someday. ? My backlog per year has around 25 games and i can't finish ? Edited October 18, 2018 by srxz 1 Quote Link to comment Share on other sites More sharing options...
srxz Posted October 19, 2018 Author Share Posted October 19, 2018 On 10/18/2018 at 3:44 AM, Scriven said: I love this plugin, one of the few I use on a daily basis. Would it be possible for the plugin to check if the game's SteamID is present as the file name? The Astonishing Game has the ID 583190. The plugin could just add store.steampowered.com/app/ and the ID to automate the process. If the game wasn't imported by the Steam import through LB, it then could just ask for the URL as it does now. I have 6500+ games I have to update manually because most games are not in the LB database while the data is right there on the Steam website and while your plugin is awesome, I think it could be even better if you added this feature. I tried the Steam import from LB last night and I might be able to do it. When LB adds the game to the Library it generates the launch parameter on Application path as: steam://rungameid/<AppId> e.g. steam://rungameid/730 So I just need to check if there's this pattern on Application path, if yes, does the scrap automatically based on the found ID, if not pops up the form to paste URL. Also I can add maybe the multiple game selection for this one. Quote Link to comment Share on other sites More sharing options...
Scriven Posted October 19, 2018 Share Posted October 19, 2018 4 hours ago, srxz said: I tried the Steam import from LB last night and I might be able to do it. When LB adds the game to the Library it generates the launch parameter on Application path as: steam://rungameid/<AppId> e.g. steam://rungameid/730 So I just need to check if there's this pattern on Application path, if yes, does the scrap automatically based on the found ID, if not pops up the form to paste URL. Also I can add maybe the multiple game selection for this one. This is great news. Can't wait to try it. ? Quote Link to comment Share on other sites More sharing options...
srxz Posted October 22, 2018 Author Share Posted October 22, 2018 On 10/19/2018 at 12:31 PM, Scriven said: This is great news. Can't wait to try it. ? Hi Scriven, you can give it a go. I added the features, let me know if its working. Quote Link to comment Share on other sites More sharing options...
Scriven Posted October 22, 2018 Share Posted October 22, 2018 4 hours ago, srxz said: Hi Scriven, you can give it a go. I added the features, let me know if its working. Seems to work great. I had a problem with the game A Detective's Novel (503820). The plugin won't download any metadata for this game. I have to check if there are others like it. But this is the greatest plugin. I selected quite a few games (maybe 50) and they all were updated in seconds. I can't imagine how long that would have taken me to do it manually. You got a Steam wishlist? I got lots of doubles and would like you to have some. ? Quote Link to comment Share on other sites More sharing options...
srxz Posted October 22, 2018 Author Share Posted October 22, 2018 5 hours ago, Scriven said: Seems to work great. I had a problem with the game A Detective's Novel (503820). The plugin won't download any metadata for this game. I have to check if there are others like it. But this is the greatest plugin. I selected quite a few games (maybe 50) and they all were updated in seconds. I can't imagine how long that would have taken me to do it manually. You got a Steam wishlist? I got lots of doubles and would like you to have some. ? That's odd, just make sure all the games that you've select are with Application Path : steam://rungameid/<idOfTheGame> , if not the multiple game option will just ignore. Another thing to try is to scrape it manually, see if it will pop the URL option or not, if it pops the application path is wrong. But in any case I will try it out as soon I get home and hope to fix it! Sure you can add me on steam! https://steamcommunity.com/id/sur0x Quote Link to comment Share on other sites More sharing options...
Scriven Posted October 22, 2018 Share Posted October 22, 2018 I think that it might not be a problem with the plugin but with the Steam website. I updated a lot of games one by one and the scraped information would suddenly stop. Maybe that will happen from time to time when updating a lot of games. 1 Quote Link to comment Share on other sites More sharing options...
CriticalCid Posted October 25, 2018 Share Posted October 25, 2018 Your plugin is awesome! Thanks so much for it. I have almost 6000 games on Steam and hundreds, if not thousands, of these games are not in the LB database, so this is quite convenient. I gave your plugin a try and wanted to share my thoughts and results with you. My test setup was a complete new LB installation were I’ve imported my whole Steam library through LaunchBox’ own Steam importer and then started to scrape all the games with your plugin. So far I have tested it with all my games from # to B (745 games). I’ve noticed that when you scrape multiple games at once, your plugin just stops continuing when it hits a game where it has problems with; for example, when the game has no Steam store page (happens when the game was removed from Steam) or if the store page has almost nothing in it like the episodes of the Back to the Future game from Telltale (https://store.steampowered.com/app/94500/). It would be nice if your plugin could just skip those problematic games and continue with the rest of the selected games instead of just stop running. On the same note it would it be nice to have some feedback if your plugin is currently doing anything or not. Right now I have to check with the Task Manager if my internet bandwidth is currently used to see if your plugin is still running or if it has stopped. I also encountered a few games where your plugin does download metadata for them but not any media files. I’ve also tried to add these games manually with your plugin and got the same issue. The following games are affected by this: https://store.steampowered.com/app/270090/ https://store.steampowered.com/app/61510/ https://store.steampowered.com/app/689910/ https://store.steampowered.com/app/522890/ https://store.steampowered.com/app/15100/ https://store.steampowered.com/app/34900/ https://store.steampowered.com/app/257350/ https://store.steampowered.com/app/228280/ https://store.steampowered.com/app/57640/ And finally I have a feature request. It would be awesome if your plugin could also scrape Flags and Tags and add them to LaunchBox as Custom Fields. Especially for ridiculously large collections like mine it’s incredible useful to be able to filter the games by that. Thanks again for your time Quote Link to comment Share on other sites More sharing options...
CriticalCid Posted October 25, 2018 Share Posted October 25, 2018 38 minutes ago, CriticalCid said: I also encountered a few games where your plugin does download metadata for them but not any media files. I’ve also tried to add these games manually with your plugin and got the same issue. The following games are affected by this: I found the issue for most of them. Your plugin does indeed download the media without any problems but LaunchBox can’t find it because they all have an apostrophe in the file name and LB doesn’t like that. If you replace the apostrophe in the file name with an underscore they show up just fine. Unfortunately I haven't figured out yet what's the problem with: //N.P.P.D. RUSH//- The milk of Ultraviolet EDIT: Also found the issue for N.P.P.D. RUSH. LB apparently wants a single underscore as substitute for multiple slashes instead of one for each slash Quote Link to comment Share on other sites More sharing options...
srxz Posted October 25, 2018 Author Share Posted October 25, 2018 5 minutes ago, CriticalCid said: I found the issue for most of them. Your plugin does indeed download the media without any problems but LaunchBox can’t find it because they all have an apostrophe in the file name and LB doesn’t like that. If you replace the apostrophe in the file name with an underscore they show up just fine. Unfortunately I haven't figured out yet what's the problem with: //N.P.P.D. RUSH//- The milk of Ultraviolet Hi @CriticalCid thanks for your testing and thoughts, I will try to fix these issues: Replace invalid characters (Im already doing this Windows wide, I block every invalid char that windows forbidden, this is a LB problem ) , this is the problem with //N.P.P.D. RUSH//- The milk of Ultraviolet , I just replace invalid chars with _, and double _ makes LB doesnt find the media, I will replace it with something else I already validate if the metadata is there, if not it just makes it null. But I guess there's some metadata going through and this breaks the plugin I will check if the game has store page, if not i will treat the exception, this should be easy Regarding the feedback this is complicated, it goes to places that Im not expert, Im not a programmer so to explain a little, I use asynchronous download, this means that you can trigger the scrape button and keep interacting with LB, this is good because it dosnt lock the interface but on the other side since it is asynchrounous I have no idea on how to get the feedback that you need. If I plan to do all the download on other way the LB will just freeze until it stops, but it will be easy to give the feedback once its finished. But, my code is always open on github, it anyone knows how to do it I will kindly approve the merge request! So just hit the scrape button and chill for now ? I will update it and release a new version maybe today, would be great if you could test it again Like I said I didnt designed this for hundreds of games so I guess I'm happy with results lol! Regarding tags, custom fields, etc. Please see this : https://gist.github.com/rodrigosiviero/f6681dbfddddf9fbcf954bbac7a33ec4 this is the information that I retrieve from Steam, which are the information that you wish to add?! Unfortunately Steam doesn't give us steam tags Any doubts or questions you can also hit me up at discord directly srxz#5034 Quote Link to comment Share on other sites More sharing options...
CriticalCid Posted October 25, 2018 Share Posted October 25, 2018 38 minutes ago, srxz said: Regarding tags, custom fields, etc. Please see this : https://gist.github.com/rodrigosiviero/f6681dbfddddf9fbcf954bbac7a33ec4 this is the information that I retrieve from Steam, which are the information that you wish to add?! Unfortunately Steam doesn't give us steam tags In your linked document is “Categories” the data that I meant with “Flags”. Regarding Tags I know that other tools are able to scrape it. For example Depressurizer: https://github.com/Depressurizer/Depressurizer I’m no developer but maybe its source code can help you figuring it out. 45 minutes ago, srxz said: Regarding the feedback this is complicated, it goes to places that Im not expert, Im not a programmer so to explain a little, I use asynchronous download, this means that you can trigger the scrape button and keep interacting with LB, this is good because it dosnt lock the interface but on the other side since it is asynchrounous I have no idea on how to get the feedback that you need. If I plan to do all the download on other way the LB will just freeze until it stops, but it will be easy to give the feedback once its finished. But, my code is always open on github, it anyone knows how to do it I will kindly approve the merge request! So just hit the scrape button and chill for now ? Gotcha. Currently it's quite tedious because your plugin constantly stops when it hits a game with no (proper) steam store page but once you've fixed this issue it should be alright. A message when the scraping is finished or ran into a problem would be neat none the less Quote Link to comment Share on other sites More sharing options...
srxz Posted October 25, 2018 Author Share Posted October 25, 2018 (edited) 13 minutes ago, CriticalCid said: In your linked document is “Categories” the data that I meant with “Flags”. Regarding Tags I know that other tools are able to scrape it. For example Depressurizer: https://github.com/Depressurizer/Depressurizer I’m no developer but maybe its source code can help you figuring it out. Gotcha. Currently it's quite tedious because your plugin constantly stops when it hits a game with no (proper) steam store page but once you've fixed this issue it should be alright. A message when the scraping is finished or ran into a problem would be neat none the less All those points are currently fixed. I just need to test them and release it, so it will break less Regarding the Depressurizer, it really scrapes the html page, I will not do that with steam, I prefer keep it "legal", what I can do is to get the categories to Custom fields and Tags from steamspy.com (see this https://steamspy.com/app/750920 ) but Ill think about it, if it really worth go to 2 apis to get information, i will double the complexity and maintainability of the scraper for a feature that only Premium members have Edited October 25, 2018 by srxz Quote Link to comment Share on other sites More sharing options...
CriticalCid Posted October 25, 2018 Share Posted October 25, 2018 21 minutes ago, srxz said: All those points are currently fixed. I just need to test them and release it, so it will break less Wow, that was fast! Good job. 21 minutes ago, srxz said: Regarding the Depressurizer, it really scrapes the html page, I will not do that with steam, I prefer keep it "legal", what I can do is to get the categories to Custom fields and Tags from steamspy.com (see this https://steamspy.com/app/750920 ) but Ill think about it, if it really worth go to 2 apis to get information, i will double the complexity and maintainability of the scraper for a feature that only Premium members have Ah, sure. I didn’t know that you only rely on the Steam API because as far as I understand does the term ‘scraping’ suggest to not need any and take the data the ‘rough’ way. And your tool is called ‘Steam Scraper’ after all I completely understand that you want to keep your plugin in completely legal terms and I’m sure that Jason is also appreciating it, so I would never ask you to do it otherwise. While I personally would really love to have the tags in LB (which would make your plugin perfect for me), I certainly don’t want to push you into a giant maintenance job by adding a second API, so I can completely understand if you would decide that it’s not worth it. Quote Link to comment Share on other sites More sharing options...
srxz Posted October 25, 2018 Author Share Posted October 25, 2018 14 minutes ago, CriticalCid said: Wow, that was fast! Good job. Ah, sure. I didn’t know that you only rely on the Steam API because as far as I understand does the term ‘scraping’ suggest to not need any and take the data the ‘rough’ way. And your tool is called ‘Steam Scraper’ after all I completely understand that you want to keep your plugin in completely legal terms and I’m sure that Jason is also appreciating it, so I would never ask you to do it otherwise. While I personally would really love to have the tags in LB (which would make your plugin perfect for me), I certainly don’t want to push you into a giant maintenance job by adding a second API, so I can completely understand if you would decide that it’s not worth it. Yeah the name of plug-in was really lazy on me ?? I will release a new version with those fix asap and then I will give a shot on the categories and tags from steamspy for the future, I just need to figure out how to put on custom fields and which way. Im accepting suggestions ? Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.