Youtube Scraper

[JoffreyV]

Nice work man! The new version 0.0.7 seems to crash my Launchbox though, even after unblocking the dll's... :-(

[srxz]

1 hour ago, JoffreyV said:

Nice work man! The new version 0.0.7 seems to crash my Launchbox though, even after unblocking the dll's... :-(

I Just released a new version, could you please try it? I have no idea whats happening, this never happened before, I mean its not a code problem I think.

[neil9000]

10 hours ago, srxz said:

I Just released a new version, could you please try it? I have no idea whats happening, this never happened before, I mean its not a code problem I think.

Both the previous version and the new one you released are working just fine for me, using Launchbox 8.3 beta 4 and windows 10 64bit fully up to date. However I do notice in that image posted that the folder is called YouTube-0.0.7 mine is just called YoutubeScraper, could that be the issue? My previous version was named like that also and worked. 

[srxz]

51 minutes ago, neil9000 said:

Both the previous version and the new one you released are working just fine for me, using Launchbox 8.3 beta 4 and windows 10 64bit fully up to date. However I do notice in that image posted that the folder is called YouTube-0.0.7 mine is just called YoutubeScraper, could that be the issue? My previous version was named like that also and worked. 

Yeah, could be it, maybe it's not handling well dashes and points, try to remove them @JoffreyV !

Glad to hear it works on the newest update @neil9000 I'm stuck for now with 8.2.

[JoffreyV]

I have also tried changing the folder name but that didn't work. Trying the 0.0.8 version now.

[JoffreyV]

After deleting the other video scraper plugin (GetSnaps from Ize) it worked. Thanks.

Edited May 14, 2018 by JoffreyV

[LiveSorcerer]

Amazing, I will follow this topic.

[fawkesyeah]

This plugin seems to be broken for certain Youtube video links.

For instance, this video will download fine (along with many others):

..But then this video will throw an Error and will not download:

Error:

Could not find the entry function for signature deciphering.

App:     LaunchBox
Version: 8.7
Type:    YoutubeExplode.Exceptions.ParseException
Site:    Void MoveNext()
Source:  YoutubeExplode

   at YoutubeExplode.YoutubeClient.<GetVideoPlayerSourceAsync>d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at YoutubeExplode.YoutubeClient.<GetVideoMediaStreamInfosAsync>d__40.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at YoutubeScraper.Youtube.<youtubeAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

Recent Log:

   10:32:50 AM Music.Prepare Start
   10:32:50 AM Music.Kill Start
   10:32:50 AM Music.Kill Finished
   10:32:50 AM Music.NotifyTrackList Start
   10:32:50 AM Music.NotifyTrackList Invoke WPF
   10:32:55 AM Music.Prepare Start
   10:32:55 AM Music.Kill Start
   10:32:55 AM Music.Kill Finished
   10:32:55 AM Music.NotifyTrackList Start
   10:32:55 AM Music.NotifyTrackList Invoke WPF
   10:32:56 AM Music.Pause Start
   10:33:01 AM Music.Prepare Start
   10:33:01 AM Music.Kill Start
   10:33:01 AM Music.Kill Finished
   10:33:01 AM Music.NotifyTrackList Start
   10:33:01 AM Music.NotifyTrackList Invoke WPF
   10:33:01 AM VlcMediaPlayer.Dispose(bool disposing) HandleManager.Remove
   10:33:01 AM VlcMediaPlayer.Dispose(bool disposing) EventManager.Detach
   10:33:01 AM VlcMediaPlayer.Dispose(bool disposing) Free
   10:33:01 AM VlcMediaPlayer.Dispose(bool disposing) ReleaseMediaPlayerFunction
   10:33:13 AM Exception

Can you please help me determine what is going on and/or how to resolve this? I am using Launchbox 8.7.

[Kondorito]

What is the URL you are pasting in the box that triggers the above error? It should not have any extra specifications (i.e. when you play a youtube video from a specific time, it adds an &timer or something similar).

 

 

[fawkesyeah]

3 minutes ago, Kondorito said:

What is the URL you are pasting in the box that triggers the above error? It should not have any extra specifications (i.e. when you play a youtube video from a specific time, it adds an &timer or something similar).

 

 

This is exactly the URL as I am pasting it in, then it errors as above: (note, the syntax of the URL is exactly the same as other, working URLs)

https://www.youtube.com/watch?v=daWPlxr4g6A

Edited September 17, 2018 by fawkesyeah
added note

[Kondorito]

Ok, I just read the whole error message; if I am not mistaken, it seems that the plugin needs to be updated, and have it added the latest yt player source code or similar; at least the error "Could not find the entry function for signature deciphering" seems to point to a parsing error. Maybe @srxz can shed some light into this.

Edited September 17, 2018 by Kondorito

[srxz]

Hey @Kondorito @fawkesyeah, thanks for the report.

 

Im at work now, I will try to look into this after I get home, it seems like youtube updated something on their side everyone is having this problem, good thing is the library that I use (YotutubeExplode) already updated I just need to compile the new version and release it.

 

[srxz]

@Kondorito @fawkesyeah could you guys please test it with 8.7 I have uploaded the new version 

 I couldn't test with the videos, i guess my license expired and video doesn't show anymore??!!? But I verified that the video throwing error got downloaded.

Also thank you @fawkesyeah for the idea to put Vive games to Launchbox, I have hundreds of them out of steam and sometimes it's just a nightmare to know what to play.

[Kondorito]

@srxz tested both options of the plugin, and got videos on both. It would be great fo @fawkesyeah to test the same scenario he previously did with the old plugin with those two videos, and see if now they are getting downloaded.

Thank you srxz for your awesome work.

[fawkesyeah]

1 hour ago, srxz said:

@Kondorito @fawkesyeah could you guys please test it with 8.7 I have uploaded the new version 

 I couldn't test with the videos, i guess my license expired and video doesn't show anymore??!!? But I verified that the video throwing error got downloaded.

Also thank you @fawkesyeah for the idea to put Vive games to Launchbox, I have hundreds of them out of steam and sometimes it's just a nightmare to know what to play.

That fixed it! At least for this specific youtube video which was not working before, is working with this update. I'll have faith for now it'll keep working, I appreciate you taking the time! ?

Also yes using Launchbox for VR/Vive/Occulus games is pretty slick. I love having the YT videos for any/all my games for getting inspired to play.

Lastly, question for you @srxz. How easy or challenging do you think it would be to make a plugin similar (or modify this one?), but for scraping Steam videos instead? Sometimes I am on a Steam game page, and grabbing the trailer from there directly is more convenient. Curious your thoughts?

[srxz]

2 minutes ago, fawkesyeah said:

That fixed it! At least for this specific youtube video which was not working before, is working with this update. I'll have faith for now it'll keep working, I appreciate you taking the time! ?

Also yes using Launchbox for VR/Vive/Occulus games is pretty slick. I love having the YT videos for any/all my games for getting inspired to play.

Lastly, question for you @srxz. How easy or challenging do you think it would be to make a plugin similar (or modify this one?), but for scraping Steam videos instead? Sometimes I am on a Steam game page, and grabbing the trailer from there directly is more convenient. Curious your thoughts?

I was actually just looking into this right now. Yes it is doable, but, on the wrong way, I would have to "scrape" the website and get information manually, this is a bad practice. Steam as far I know doesn't provide you API to get metadata from their games.

Actually I was looking to get ALL the metadata from steam and put it on Game Details of launchbox, such as publisher, developer, year, etc. The process to get only the video is the same, I need to find thqe webpage, crawl until I find the file and download it, there's some problems with this, if steam changes the page it will break and steam can easily block me if I request too much.

 

[fawkesyeah]

That's a great idea. You think Steam might block? Is it parsing as if it were a web browser, if so then how could they block you?

I wonder if you could grab certain details from the Steamdb.info, which does not seem to have Videos and perhaps other details, but as a backup?

Personally, the most important details to scrape would be the video, photos, short description, possibly genre and date released. However if all details are possible then of course that.

Update: Just found these two links, I wonder if they would help:

https://github.com/SteamRE/SteamKit

https://wiki.teamfortress.com/wiki/User:RJackson/StorefrontAPI

https://github.com/prncc/steam-scraper

Edited September 18, 2018 by fawkesyeah

[srxz]

1 hour ago, fawkesyeah said:

That's a great idea. You think Steam might block? Is it parsing as if it were a web browser, if so then how could they block you?

I wonder if you could grab certain details from the Steamdb.info, which does not seem to have Videos and perhaps other details, but as a backup?

Personally, the most important details to scrape would be the video, photos, short description, possibly genre and date released. However if all details are possible then of course that.

Update: Just found these two links, I wonder if they would help:

https://github.com/SteamRE/SteamKit

https://wiki.teamfortress.com/wiki/User:RJackson/StorefrontAPI

https://github.com/prncc/steam-scraper

 

Last night I spent trying to do a crawler myself on C#, since I never worked with c# before it took me a long time to do it, but I managed to get the Game logo, developer, publisher, release date and video. I don't think we need more stuff correct? Maybe I can get more images but I have no idea how to add it to Launchbox programmatically, I just need now to put the code into a new plugin, can't use the same as Youtube.

I was thinking in do the same as the Youtube URL, you paste the URL link and it parses everything.

Steam might block if you try to scrape like 200 games at once, so I'm thinking if I add the bulk action it would be bad, but, I don't think that someone has like Mame 5000 games so one each time would be better.

Regarding those like, thanks I found them yesterday and they are not good for app metadata, except this one https://wiki.teamfortress.com/wiki/User:RJackson/StorefrontAPI , This was a good find I think I can parse this and get all the info that we need without scrape everything!

I'm a little bit f*** on my work but I will try to code something on the following weeks, if you are on Discord please DM me so I can send you the first versions as soon they are done since I do not have premium anymore would be good someone testing it.

 

 

[fawkesyeah]

I agree, a bulk feature doesn't seem necessary, at least at first. Maybe later it's feasible to add a bulk feature that limits itself somehow, i.e. only processes 20 per hour (ish). But having a plugin that scrapes games individually I believe would be higher priority.

When you mentioned Game logo, are you talking about the Steam banner? If not, the banner is certainly important too. Personally I use Banners as the primary view for Windows based games. Grabbing them manually has been a huge hassle (one that I've tried to alleviate with WinAutomation). I'm most excited about the Banner and the Video ?

Other than the banner, maybe Genre(s)? That all sounds like an excellent foundation for a plugin.

I understand, life comes first. What's your discord 4 digit tag? I'll add you. Mine is #4024 (same username as I am here)

Edited September 18, 2018 by fawkesyeah

[srxz]

50 minutes ago, fawkesyeah said:

I agree, a bulk feature doesn't seem necessary, at least at first. Maybe later it's feasible to add a bulk feature that limits itself somehow, i.e. only processes 20 per hour (ish). But having a plugin that scrapes games individually I believe would be higher priority.

When you mentioned Game logo, are you talking about the Steam banner? If not, the banner is certainly important too. Personally I use Banners as the primary view for Windows based games. Grabbing them manually has been a huge hassle (one that I've tried to alleviate with WinAutomation). I'm most excited about the Banner and the Video ?

Other than the banner, maybe Genre(s)? That all sounds like an excellent foundation for a plugin.

I understand, life comes first. What's your discord 4 digit tag? I'll add you. Mine is #4024 (same username as I am here)

@fawkesyeahBanner I mean the game logo, for example:

 

https://steamcdn-a.akamaihd.net/steam/apps/764790/header.jpg

 

from https://store.steampowered.com/app/764790/The_Messenger/

 

is that it?