Update suggestion: update if metadata is different

[Theman8631]

 Currently there  is the option to download and replace all meta-data fields,  and there is an option to download the fields that are empty.

 It would be nice if there was an option to update fields that have been changed since last updated. I can imagine that there could be fast “is this the same?” Hash check that should be faster then blindly redownloading everything.

Edited May 29, 2019 by Theman8631

[neil9000]

16 minutes ago, Theman8631 said:

 Currently there  is the option to download and replace all meta-data fields,  and there is an option to download the fields that are empty.

 It would be nice if there was an option to update fields that have been changed since last updated. I can imagine that there could be fast hash check that ultimately would be faster in processing then redownloading completely or a behind-the-scenes “change number” where  each digit represented one of the fields and whenever a change was propagated on the database it  could increase its representative digit by one.   When the update check happens for the launch box ID associated it checks each individual digit and only updates the fields were the digits are not identical or some such.   I’m sure there’s many ways to handle this but that feels processing wise pretty efficient.

Interesting idea, but i'm not sure how much something like that is needed to be honest. The entire metadata.xml file is only around 175MB and is downloaded in its entirety when you prompt to update metadata. 175MB isnt a particularly large file these days with the speeds of most peoples internet, it would probably take longer to parse what is new information than it would to download the entire metadata file.

Just my personal opinion though.

[Theman8631]

Just now, neil9000 said:

Interesting idea, but i'm not sure how much something like that is needed to be honest. The entire metadata.xml file is only around 175MB and is downloaded in its entirety when you prompt to update metadata. 175 isnt a particularly large file these days with the speeds of most peoples internet, it would probably take longer to parse what is new information than it would to download the entire metadata file.

Just my personal opinion though.

What about the images, music and movies though?  If a box front image is set but but it sucks and has been improved I am still in the situation of wanting to update it. 

[neil9000]

3 minutes ago, Theman8631 said:

What about the images, music and movies though?  If a box front image is set but but it sucks and has been improved I am still in the situation of wanting to update it. 

OK, yeah images i'm not to sure about to be honest, i know the metadata is a single file that is stored local on your machine and is replaced when you search for metadata, when is says its updating metadata its not scanning games just downloading a whole new file.

[Theman8631]

So I guess the real meat of what Im hoping for is to update both metadata and media in this way without redownloading everything 

[Jason Carr]

Per metadata, the download is much, much smaller due to compression (30 megabytes rings a bell), so I don't think the metadata is an issue at all. One reason why we do it that way is because it saves CPU time on the server. We get a lot of traffic and if the metadata downloads started requiring CPU activity, it could bring the server to its knees.

But for images, the problem becomes how do you tell whether an image is an updated (better) image and can replace an existing one, or if it's a separate image that should be added to your collection. The short answer is we can't. We do use checksums to determine whether images are the same, but we don't have any magic image processing code that helps us to determine similar images. We'd probably have to license the moon from Google in order to get something like that.

[Theman8631]

17 minutes ago, Jason Carr said:

Per metadata, the download is much, much smaller due to compression (30 megabytes rings a bell), so I don't think the metadata is an issue at all. One reason why we do it that way is because it saves CPU time on the server. We get a lot of traffic and if the metadata downloads started requiring CPU activity, it could bring the server to its knees.

But for images, the problem becomes how do you tell whether an image is an updated (better) image and can replace an existing one, or if it's a separate image that should be added to your collection. The short answer is we can't. We do use checksums to determine whether images are the same, but we don't have any magic image processing code that helps us to determine similar images. We'd probably have to license the moon from Google in order to get something like that.

Well we can get a good assumption that doesn’t take much processing by comparing an images file size and dimension.  Because these fluctuate so much on the gamesdb I will guarantee 99% of the time a change in artwork will change dimensions and size, which are both characteristics of the files properties and should be able to be referenced or logged with very little processing and could be downloaded in a similar images xml and processed client side.  

Since it would be the preferred option to redownloading everything redundantly (which Im sure Im not the only one to want updated images) the net bandwidth it would save server side seems worth the thought for a possible bandwidth advantage  and drastically lowering individual requests per image.

Edited May 29, 2019 by Theman8631

[Theman8631]

Simpler yet- when a file is replaced on the gamesdb timestamp the filename server side example:

New front image accepted on 5-29-2019 12:38

Front529191238.jpG

That would be all the processing change needed server side - your doing some processing to save it in the db anyway - and just change the current scan to check if a filename is the same before downloading.  If its the same skip, now that should be way more efficient all around and way less bandwidth!

As it is now replace everything even if its the same thing is a big load of unnecessary processing and bandwidth on both sides.

Edited May 29, 2019 by Theman8631

[Jason Carr]

@Theman8631 We already do calculate in checksums, which by nature already handles file size and dimensions. That's already being used. If you tell it to only update and not replace, it won't download anything that matches checksums and sizes.

[Theman8631]

1 minute ago, Jason Carr said:

@Theman8631 We already do calculate in checksums, which by nature already handles file size and dimensions. That's already being used. If you tell it to only update and not replace, it won't download anything that matches checksums and sizes.

Ok... but I WANT it to replace images that do not match checksums.  That’s very specifically what I am suggesting.  Currently there is no option to update images, right?  

 

[Jason Carr]

7 minutes ago, Theman8631 said:

Ok... but I WANT it to replace images that do not match checksums.  That’s very specifically what I am suggesting.  Currently there is no option to update images, right?  

If an image does not match a checksum, then you have absolutely no idea if it's a different image or not. What do you want to do in that scenario? Just throw out the old image and trust that the new one is better? Images on the games database aren't often deleted, anyways. Most often, new images are added in addition to the ones already there, so deleting existing images wouldn't really do much for us.

[Theman8631]

10 minutes ago, Jason Carr said:

If an image does not match a checksum, then you have absolutely no idea if it's a different image or not. What do you want to do in that scenario? Just throw out the old image and trust that the new one is better? Images on the games database aren't often deleted, anyways. Most often, new images are added in addition to the ones already there, so deleting existing images wouldn't really do much for us.

 The odds are very likely it’s a new image,  and yes,  the option that I think would be useful is to download new images from the database, if my pictures are likely different.   The images would be better because that’s what happens when images are replaced on the games database,  they are replaced with more appropriate, more accurate pictures.   The frequency that this is not the case is far more rare than when a new image is posted.   When a new image is posted it has a much higher likelihood of being inaccurate or low quality,  then times that someone replaces an image on the database,  because they are consciously doing so with the intention that they are improving upon it.  However as it is now when they do so nobody gets those images unless they choose to overwrite all of the images,  including images where it is the same image.

Another way to think about this - why is there an option for me to override my current images with the same image?  Can we have an option in the future to overwrite the images that are different from the images on the gamesdb.

Edited May 29, 2019 by Theman8631

[Jason Carr]

4 minutes ago, Theman8631 said:

 The odds are very likely it’s a new image,  and yes,  the option that I think would be useful is to download new images from the database, if my pictures are likely different.   The images would be better because that’s what happens when images are replaced on the games database,  they are replaced with more appropriate, more accurate pictures.   The frequency that this is not the case is far more rare than when a new image is posted.   When a new image is posted it has a much higher likelihood of being in accurate or low quality,  then times that someone replaces an image on the database,  because they are consciously  doing so with the intention that they are improving upon it.

I see. So you want to be able to delete existing images in your collection that were deleted from the games database and replace them with new images that were updated. Currently once images are deleted from the games database, we no longer track them, so we can't really identify them after the fact, so we would have to make some significant changes to get that to work. I can see the value in that though. Thanks for clarifying it for me.

[Theman8631]

You could just have an option to replace the images that dont match the hash.  That isn’t an incredibly significant change.  Your already checking, and people who want to update their library with newer images redownload every image for every game.  There is an option to redownload the images I already have and it would be faster and less bandwidth om both sides for an option to redownload the images that dont match the image hash on the db.  

If the hash is checking why does it redownload the thing you have... 

Edited May 29, 2019 by Theman8631

[Jason Carr]

2 minutes ago, Theman8631 said:

You could just have an option to replace the images that dont match the hash.  That isn’t an incredibly significant change.  Your already checking, and people who want to update their library with newer images redownload every image for every game.  There is an option to redownload the images I already have and it would be faster and less bandwidth om both sides for an option to redownload the images that dont match the image hash on the db.  

Alright, so first off, people who want to update their library with newer images do not have to redownload every image for every game. That's the point of choosing to update without replacing. You get all the new images without having to re-download anything.

I understand though that you want an option that simply clears out all existing images for a category if the checksums don't match. It's not quite that simple since there's often multiple images for each category, but I get your request.

[Theman8631]

1 minute ago, Jason Carr said:

Alright, so first off, people who want to update their library with newer images do not have to redownload every image for every game. That's the point of choosing to update without replacing. You get all the new images without having to re-download anything.

I understand though that you want an option that simply clears out all existing images for a category if the checksums don't match. It's not quite that simple since there's often multiple images for each category, but I get your request.

When you say “people can update new images without redownloading every image” I just want to end on the final point, they can’t.  You said “thats the point of downloading without replacing” but unless I’m mistaken the point of downloading without replacing is to download images for image types you dont have.  So therefore, you can’t update to the new images unless you happen to have no image of that image type without downloading and replacing all images.

[Jason Carr]

1 minute ago, Theman8631 said:

When you say “people can update new images without redownloading every image” I just want to end on the final point, they can’t.  You said “thats the point of downloading without replacing” but unless I’m mistaken the point of downloading without replacing is to download images for image types you dont have.  So therefore, you can’t update to the new images unless you happen to have no image of that image type without downloading and replacing all images.

You're mistaken.

[Theman8631]

4 minutes ago, Jason Carr said:

You're mistaken.

Oh I see so updating without replacing metadata will download a new image if I have that image type but I have an older version of the image from the database?

Edited May 29, 2019 by Theman8631

[Jason Carr]

Just now, Theman8631 said:

Oh I see so updating without replacing metadata will download a new image if I have that image type but I have an older version of the image from the database?

Yes. You'll end up with two images.

[Theman8631]

Oh then there may be a typo.  It says “Launchbox will update metadata and media, but only for empty fields or missing media.”   Are you saying that it actually functions that it will “update metadata and media,  but only for empty fields and will add new media.” 

However for something like box front, theres only one box front that I see per game when using that option and it doesn't download a new box front from what I can tell...