Vidsnap Scraper

[maikeru1986]

Vidsnap Scraper

View File

---

DISCLAIMER: Use at your own risk! Please read and understand Youtube's terms of service. I will not be held liable for any mis-use of any product or service.

 

This plugin makes use of the open source and unlicensed application, youtube-dl yt-dlp (forked from youtube-dl) to download game videos.

It first downloads videos from the link provided in the metadata from LaunchBox, skipping videos longer than a specified duration, and then searching for videos for missing games based on input search terms.

It is intended for Modern Windows games as this was my need initially. It may work on other platforms but is not well tested.

 

Installation Instructions:

Note:  For this plugin to work, you will need to have yt-dlp.exe (included in .zip). Extract the contents of the .zip folder and place it in your LaunchBox plugins folder.

 

Usage Instructions:

Select a game or multiple games, right-click and select "Download Video Snap."

Adjust the maximum duration if desired.

The "Get Missing videos" checkbox sets whether or not to search for videos of games that don't have URL's in their metadata.

The "Trim videos" checkbox sets whether or not to trim videos after it has been downloaded.

Enter desired search terms for Missing videos.

click "Start" to start the download process.

Games will download asynchronously. resuming downloads is also supported natively.

 

Known Issues:

probably many that I am unaware of.

 

Other:

PLEASE NOTE: I AM NOT A DEVELOPER! I know very little about C# and programming in general, thus probably wont be able to address many issues that you may face. That being said I will try and fix things were I can. I've provided the spagetti source code for anyone to modify, improve, etc. (it seems you need to unblock the forms in the "Resources" folder -> RMB->properties->unblock) 

As of now, i'm more comfortable with the usability of the plugin, however there are still probably many issues, any comments to provide insight/improvement to the code etc. would be much appreciated.

Thanks to @srxz for his source of Youtube Scraper of which this is based.

 

Thanks: 

@srxz @JoeViking245 @C-Beats

 

maikeru86

 

 

---

• Submitter
  maikeru1986
• Submitted
  12/16/2020
• Category
  Third-party Apps and Plugins

 

Edited July 20, 2021 by maikeru1986
added disclaimer

[BZZ]

Well just downloaded it and didnt expect much as I used Youtube Scraper for about 30 minutes before deleting it. Didnt work probably due to version updates rather than plugin problems.

 

But on the topic this one works perfect. Downloads fast and correct(at least so far) files.

[OdinsPlayground]

I've been looking for something like this. But any chance you could apply a video time crop? For example: the plugin / search might find a 5min video, but crops it to 30 sec if the settings is set to crop to that. Would make it a lot more useful, instead of downloading lengthy gameplay videos. As well as higher compatibility in finding videos that can be used. 

[gzubsc]

Hey! The plugin wasn't working anymore.

I just had to update the youtube-dl.exe file on the plugin directory by this one (v2021.04.07 FI) and all was fine again ?

[maikeru86]

On 1/19/2021 at 1:29 AM, OdinsPlayground said:

I've been looking for something like this. But any chance you could apply a video time crop? For example: the plugin / search might find a 5min video, but crops it to 30 sec if the settings is set to crop to that. Would make it a lot more useful, instead of downloading lengthy gameplay videos. As well as higher compatibility in finding videos that can be used. 

Hey there, for some reason I missed this post, I think I have two accounts...

Anyhow, I was thinking about this initially by maybe using ffmpeg or something. I'll maybe relook at that idea.

No promises it will be anytime soon though, as I'm quite busy at work at the moment.

[maikeru1986]

On 1/19/2021 at 1:29 AM, OdinsPlayground said:

I've been looking for something like this. But any chance you could apply a video time crop? For example: the plugin / search might find a 5min video, but crops it to 30 sec if the settings is set to crop to that. Would make it a lot more useful, instead of downloading lengthy gameplay videos. As well as higher compatibility in finding videos that can be used. 

I've put together something that kinda works as a start. please try v0.3.0:

 

[OdinsPlayground]

Thanks!

Just did some testing of it. I trimmed them all, some worked fine and some have a dead 3-4 sec at the beginning of the video where only sound plays. If it wasn't for this, I'd say it's quite successful at what it's meant to do. I added a video as an example. I also experienced it getting stuck on a video download, where I just had to cancel.

I know this is probably way too much, but in the future I would have loved to be able to preview video, pick the right one and set manual adjustments (start and end) for each of them. Would be a bit more work on the user end when doing that, but would assure the proper video clip and part is downloaded. 

 

[maikeru1986]

Yeah there's definitely some bugs I need to work out or at least some learning to do with FFmpeg.

• I think the dead seconds is the part between the closest previous keyframe and the cut position. FFmpeg supposedly uses an edit list to instruct the player to start from the cut position, but unfortunately not all players support edit lists.
• I also noticed it getting stuck on some videos sometimes, which hasn't heppened before, but sometimes that same video would complete in another run. I didn't change anything else relating to the downloading part  however I did notice once FFmpeg is in the same folder as youtube-dl it allows downloading of higher quality/different formats.

I'll look into these issues as soon as I have a bit more free time, which unfortunately is hard to come by at the moment.

With regards to previewing videos etc. I doubt that would be anytime soon, or at all, sorry. Besides being a significant developement to the current, theres also the issue of the plugin potentially "scraping" 100's of videos which could lead to issues with google/youtube putting up capchas or IP bans... who knows. I guess I should put a disclaimer in the original post. But who knows, I might look into it one day as that would be kinda cool. 

[C-Beats]

Yeah, when using FFmpeg and simply copying the video codec you can only cut from I-Frame to I-Frame, to get around this and cut wherever you want you have to use a video codec and let the program re-encode the video with it. I do this using other tools usually (AviDemux is my go to) and use x264 encoding with a mp4 container.

[OdinsPlayground]

5 hours ago, maikeru1986 said:

Yeah there's definitely some bugs I need to work out or at least some learning to do with FFmpeg.

• I think the dead seconds is the part between the closest previous keyframe and the cut position. FFmpeg supposedly uses an edit list to instruct the player to start from the cut position, but unfortunately not all players support edit lists.
• I also noticed it getting stuck on some videos sometimes, which hasn't heppened before, but sometimes that same video would complete in another run. I didn't change anything else relating to the downloading part  however I did notice once FFmpeg is in the same folder as youtube-dl it allows downloading of higher quality/different formats.

I'll look into these issues as soon as I have a bit more free time, which unfortunately is hard to come by at the moment.

With regards to previewing videos etc. I doubt that would be anytime soon, or at all, sorry. Besides being a significant developement to the current, theres also the issue of the plugin potentially "scraping" 100's of videos which could lead to issues with google/youtube putting up capchas or IP bans... who knows. I guess I should put a disclaimer in the original post. But who knows, I might look into it one day as that would be kinda cool. 

@maikeru1986 No worries regarding the "request", more of an "This would be awesome in the future"-idea, but obviously something that would require a lot of work. Not expecting anything. Definitely fixing the timeframe bug would be the biggest actual request, so it's in a workable state. 

Would the AviDemux work to fix it as C-Beats mentioned? Using that codec instead? I see they have dev tools and it's open source. Thanks for the suggestions @C-Beats, been looking for a Windows app to do that. 

Edited July 20, 2021 by OdinsPlayground
Tagged

[maikeru1986]

Yes, @C-Beats is right. The problem is I opted to copy the videos instead of re-encode them. I did this because copying is much quicker (tbh I didn't test re-encoding to see the how long it would take). I'll do this as a quick fix in the mean time (at least I think it should work).

I'll also check out AviDemux. However what I would like ultimately is to only download a portion of a video.

[maikeru1986]

On 7/20/2021 at 7:27 PM, OdinsPlayground said:

@maikeru1986 No worries regarding the "request", more of an "This would be awesome in the future"-idea, but obviously something that would require a lot of work. Not expecting anything. Definitely fixing the timeframe bug would be the biggest actual request, so it's in a workable state. 

Would the AviDemux work to fix it as C-Beats mentioned? Using that codec instead? I see they have dev tools and it's open source. Thanks for the suggestions @C-Beats, been looking for a Windows app to do that. 

minor update as I said I would earlier. Tested it on 55 games, downloaded and trimmed 52/55, but that's not necessarily the trimming (I actually think it just didn't find those videos).

Trimming takes much longer though as it has to re-encode the video.

 

[OdinsPlayground]

Just tested and it does the job! Don't mind it taking a bit longer to convert, as long as it's working. 

Bug / inconvenience: Getting 50++ pop-ups I have to close individually after parsing.  It's every game it doesn't find.  I think the list given at the end of which games didn't download is enough. 

[maikeru1986]

14 hours ago, OdinsPlayground said:

Just tested and it does the job! Don't mind it taking a bit longer to convert, as long as it's working. 

Bug / inconvenience: Getting 50++ pop-ups I have to close individually after parsing.  It's every game it doesn't find.  I think the list given at the end of which games didn't download is enough. 

Oh I'll take that out no problem. It was for testing purposes mainly. 

[maikeru1986]

4 hours ago, maikeru1986 said:

Oh I'll take that out no problem. It was for testing purposes mainly. 

commented out those messages, didn't actually test it but fairly sure it should work. Let me know if it doesn't. I'll keep in mind for a future version maybe to write an error log.

[darreldearth]

Awesome! only thing that could make it better would be to add an option to paste in a specific URL to a chosen Youtube video

Also, seems that the plugin downloads the same video regardless of what keywords i use. Seems like its stuck on using the Trailer keyword. Only way i can get it to download something different is to lower the maximum length from 600 to something like 200

Edited December 13, 2021 by darreldearth

[maikeru1986]

On 12/13/2021 at 2:29 AM, darreldearth said:

Awesome! only thing that could make it better would be to add an option to paste in a specific URL to a chosen Youtube video

Also, seems that the plugin downloads the same video regardless of what keywords i use. Seems like its stuck on using the Trailer keyword. Only way i can get it to download something different is to lower the maximum length from 600 to something like 200

That's interesting, I'll have a look into the trailer thing when I get a chance and get back to you . with regards to the link, that should be easy to do but not what I was going for with the plugin as if you are already going for a specific link manually then it might be easier just to use youtube-dl or yt-dlp to do that? alternatively, you could paste in the video url textbox before running the plugin.

nevertheless, I'll see what I can do.

Edited December 20, 2021 by maikeru1986

[maikeru1986]

On 12/13/2021 at 2:29 AM, darreldearth said:

Awesome! only thing that could make it better would be to add an option to paste in a specific URL to a chosen Youtube video

Also, seems that the plugin downloads the same video regardless of what keywords i use. Seems like its stuck on using the Trailer keyword. Only way i can get it to download something different is to lower the maximum length from 600 to something like 200

1) I had a look at the above issue, but I can't seem to completely replicate it and also I'm thinking that I wouldn't be able to fix it. 

It could be the way ytsearch function works in yt-dlp,but it's not as well documented as the other functions and I'm not sure which parameters to change. However, it could possibly be youtube itself giving you the same video regardless of search terms (I don't know how youtube algorythms work)???

I did change some parameters and have uploaded a new version, but I doubt it changed anything at all.

Below are some screenshots:

 

2) I thought about adding a custom URL link, but would then only be for 1 video at a time, hence I do suggest just pasting in the video URL in the edit menu of the game before running the plugin. i'll add it to the todo list and have a look again when I get some free time.

 

Sorry I wasn't much help on this one.

[JoJoJuJu59]

This didn't work for me (v0.47).  I'm on v12.11 of LB.  Putting the new builds of Vidsnap Scraper into my Plugins folder completely crashes/hangs Launchbox at the splash-screen so I had to go back to an old version of Vidsnap Scraper that was in an earlier beta form but with missing features.  Would love to try the newer builds with added features because it's one of the more useful plugins for me and I appreciate the author's work.

[maikeru1986]

On 5/15/2022 at 5:05 PM, JoJoJuJu59 said:

This didn't work for me (v0.47).  I'm on v12.11 of LB.  Putting the new builds of Vidsnap Scraper into my Plugins folder completely crashes/hangs Launchbox at the splash-screen so I had to go back to an old version of Vidsnap Scraper that was in an earlier beta form but with missing features.  Would love to try the newer builds with added features because it's one of the more useful plugins for me and I appreciate the author's work.

Sorry only seeing this now. I'll have a look.

Edit:

I think i'll need some more info, not sure what, but it seems to be working on my end with LB v12.11 and Vidsnap Scraper v0.47 (I downloaded it from here again), see below. Plugins can definitely make LB hang though, whats the latest version that worked for you?

Edited July 19, 2022 by maikeru1986