Library Brainstorming

[Jason Carr]

Great discussion guys. Pogowolf, I'd love to be able to define a standard to include metadata and images, etc., inside of a ROM file, but unfortunately there's no way to do it without breaking support for all of the emulators. The only way to do it would be to get all of the emulator developers on board, which would of course be impossible. Once you change the structure of a file at all, for the most part it will no longer work with existing software. We will be looking into which emulators to include soon; I'll definitely take a look at the ones you've mentioned.

[bd00]

Does he mean a container native to LaunchBox? That holds metadata, images, rom, etc. When LB opens it will still pass the normal rom file to the emulator.

[Jason Carr]

Ah, sure, that would make sense. Though I think I'd rather use a metadata file per previous discussion. Bd, you'll notice that the new export process does automatically arrange all the games for the zip file, so I'm not really far off from an option to "consolidate game files". At this point I didn't add the metadata files, but that wouldn't be too hard. Just didn't yet see that compelling of a reason to do it at this point. Could also move the images into the game folder, which would make it an isolated set. Think that's worth doing?

[SentaiBrad]

Jason Carr said Ah, sure, that would make sense. Though I think I'd rather use a metadata file per previous discussion. Bd, you'll notice that the new export process does automatically arrange all the games for the zip file, so I'm not really far off from an option to "consolidate game files". At this point I didn't add the metadata files, but that wouldn't be too hard. Just didn't yet see that compelling of a reason to do it at this point. Could also move the images into the game folder, which would make it an isolated set. Think that's worth doing?

I personally think there is such a thing as over organizing. I'm not saying dump it all in a folder and go, but should it be complicated? I think an images folder and an XML file are fine enough. If you wanted to maybe split it all up by system? Though, I don't keep any of my games in the games folder, I just use LB as is in a folder and its points to everywhere else, so honestly I don't care how the data is stored. I just want to make sure there is no chance of it breaking, which the way it is now is hard to break unless you start deleting files.

[Neon32]

Jason Carr said We will be looking into which emulators to include soon; I'll definitely take a look at the ones you've mentioned.

does this mean that you will be using these emulators as a core for LB like in Openemu or just including it the package so that user can use it if he wants or not use it ?

[bd00]

Jason Carr said Ah, sure, that would make sense. Though I think I'd rather use a metadata file per previous discussion. Bd, you'll notice that the new export process does automatically arrange all the games for the zip file, so I'm not really far off from an option to "consolidate game files". At this point I didn't add the metadata files, but that wouldn't be too hard. Just didn't yet see that compelling of a reason to do it at this point. Could also move the images into the game folder, which would make it an isolated set. Think that's worth doing?

Yes I agree, just trying to determine what he meant.

SentaiBrad said I personally think there is such a thing as over organizing. I'm not saying dump it all in a folder and go, but should it be complicated? I think an images folder and an XML file are fine enough. If you wanted to maybe split it all up by system? Though, I don't keep any of my games in the games folder, I just use LB as is in a folder and its points to everywhere else, so honestly I don't care how the data is stored. I just want to make sure there is no chance of it breaking, which the way it is now is hard to break unless you start deleting files.

I agree, organizing can be taken too far to the point where data actually becomes less easily accessible. However, I still would like the images to be stored in a folder with the ROM file along with other things such as game music/manuals etc. all content relating to that game. Then a local scraper added to LaunchBox, that finds all this content in the games folder and adds it to the library. As it stands, images are over here, game manuals are over there, game configs & saves are here, game music is there. In the end I had remove all my game files away from all the other content to make adding games to LB bearable. I originally had everything organized in a logical, tidy and accessible manner, now it is all fractured. I have however fully committed to LaunchBox. I have all my ROM files organized in a ROM folder in the LB directory, emulators in an emulator directory, etc. I have even moved over Xpadder. I am now at the point where everything resides within the LB directory and it is fully portable. I even updated all my scripts so they work in a portable environment. I now sync this folder to other systems around the house, which all update automatically when I make any changes/additions. Joystick profiles, per-game configs, save games, etc. are all shared, so once set up, that is it for all systems and I start a game in on one system and save state it and continue in another room. I have set up the split platform libraries like you can see in the automation script I posted on another thread, which provides a better menu system for the HTPC's and improved performance for the less powerfull systems. The only problem I have is PC games, for obvious reasons. So i have a LB insatance outside of the main directory that is not synced between systems, but specific to that system. LB still links to that instance, but THAT instance is is different on each system. Not a major problem. Also, I have 4 brothers and next time I go home I will be taking the library on a HDD and setting it up on their systems. It is just a case of copying over the files and it is ready to go. It also makes it super easy to back everything up. The only problem is that if I ever had to set up LaunchBox again, all the previous scraped data is redundant. All my manual edits are gone. Everything needs to be scraped from TheGamesDB again and I have to add all my manual edits from scratch. This is why I urge the local scraper and better organization of other content. Also if I ever changed from LB to a different frontend, I would have to reorganize all my data again and everything would need scraping and setting up again. Think of the way XBMC does it. It scrapes the media's folder, looking for all content associated with the video. Images, nfo's, music, trailers, etc. Anything it doesn't find it then proceeds to scrape from the net. Once you have all the data in the directory, it is much faster to set up XBMC a second time. Or even set up a different media center, because your not left feeling like a prisoner - "It is too much work to reorganize everything, plus I will have to download EVERYTHING again."

[bd00]

Conehead said

Jason Carr said We will be looking into which emulators to include soon; I'll definitely take a look at the ones you've mentioned.

does this mean that you will be using these emulators as a core for LB like in Openemu or just including it the package so that user can use it if he wants or not use it ?

Maybe start by allowing LB to download emulators (from official source), unzipping them, storing them in a emu directory and linking the platform to them. Maybe even set some default command-line options (which can be changed by the user). Or create your own repository. This way, users don't have to have what they don't want (like your drop down list of platforms). Will keep your .exe size down too.

[pogowolf]

Jason Carr said Great discussion guys. Pogowolf, I'd love to be able to define a standard to include metadata and images, etc., inside of a ROM file, but unfortunately there's no way to do it without breaking support for all of the emulators. The only way to do it would be to get all of the emulator developers on board, which would of course be impossible. Once you change the structure of a file at all, for the most part it will no longer work with existing software. We will be looking into which emulators to include soon; I'll definitely take a look at the ones you've mentioned.

Like Bd00 stated.. not if it's local for FB. for 'standard' roms this wouldn't be too much of an issue.. but I do see problems with DosBox, Scrumm, MAME, etc. WORSE case for like a 2600.. you just allow the user to export the rom image from the CART file... which would take care of issue of other emulators not using the CART format.. until they jump on board. another option.. would be to keep a metabase of the hash for each rom image. (like what the GoodTools do) and link the hash with the game.. this way you don't need to even worry about the game itself. still thinking about that idea to figure out if it's good or not.. LOL

[pogowolf]

SentaiBrad said

Jason Carr said Ah, sure, that would make sense. Though I think I'd rather use a metadata file per previous discussion. Bd, you'll notice that the new export process does automatically arrange all the games for the zip file, so I'm not really far off from an option to "consolidate game files". At this point I didn't add the metadata files, but that wouldn't be too hard. Just didn't yet see that compelling of a reason to do it at this point. Could also move the images into the game folder, which would make it an isolated set. Think that's worth doing?

I personally think there is such a thing as over organizing. I'm not saying dump it all in a folder and go, but should it be complicated? I think an images folder and an XML file are fine enough. If you wanted to maybe split it all up by system? Though, I don't keep any of my games in the games folder, I just use LB as is in a folder and its points to everywhere else, so honestly I don't care how the data is stored. I just want to make sure there is no chance of it breaking, which the way it is now is hard to break unless you start deleting files.

it's easy to break. Just move the files. best case for something like that would be one XML per game. though really, the XML could be used just for additional information.. the standard meta data shold be in a database. and Using external images, xml, data, what have you does allow for leaving 'poop' all over the drive so you would need to add a clean up util to make verify the information. this way everything is self contained. Not saying it's the best option.. only an option.

[Vinicius256]

Hey Jason, I made a mock-up based on the OpenEmu style with the ''Charms bar'' idea from bd with a kinda ''Modern'' look, tell me what you guys think:

Spoiler

Also, there's these ''TV Mode'' mock-ups I made just for fun, in case Jason makes a feature like Steam Big Picture someday and needs some ideas:

Spoiler

Spoiler

Spoiler

That's it, opinions/thoughts are welcome

[pogowolf]

Vinicius256 said Hey Jason, I made a mock-up based on the OpenEmu style with the ''Charms bar'' idea from bd with a kinda ''Modern'' look, tell me what you guys think:

Spoiler

personally I like the first one better than anything.. however, even having the list of consoles.. you could have 30+ consoles.. listing like this (for large collections) I don't believe will work in the long run. my thoughts would be to have a fly out (kinda like the current filters) but filter the list of games by genre and/or keyword.. from my experance people don't tend to think "I'm going to play a PS2 game".. then think.. "I want a racing game".. or "I'm going to play mario" .. *shrug* but then I like to look though my list sometimes.. But that's why I like the current LB interface. it'll list everything.

[SentaiBrad]

pogowolf said

Vinicius256 said Hey Jason, I made a mock-up based on the OpenEmu style with the ''Charms bar'' idea from bd with a kinda ''Modern'' look, tell me what you guys think:

Spoiler

personally I like the first one better than anything.. however, even having the list of consoles.. you could have 30+ consoles.. listing like this (for large collections) I don't believe will work in the long run. my thoughts would be to have a fly out (kinda like the current filters) but filter the list of games by genre and/or keyword.. from my experance people don't tend to think "I'm going to play a PS2 game".. then think.. "I want a racing game".. or "I'm going to play mario" .. *shrug* but then I like to look though my list sometimes.. But that's why I like the current LB interface. it'll list everything.

But there would be a "list all" option and by default it would do this unless you chose to save your filter selection when you close LB. The way it works wouldn't change, just how it looks.

[pogowolf]

Ok.. let's see if I can't write convey the 'issue' I'm trying to solve. Not trying to be a donkey here, nor ruffle any feathers. I'm writing things because it helps me think. Issue: Keeping the meta data downloaded from 'somewhere' (in this case TheGamesDB.net) associated with the game itself. #1 - They way it's done currently is that all the meta data is downloaded and saved in one XML file. (mine is currently 78 meg) which is parsed on the fly for information, possibly partial saved on update, and re-read into LB after each edit. Pros: 1) Keeps the system 'clean' and it easy to access data 2) Keeps the data in once place, making the whole of LB more portable. Con: This process slows as the size of the XML file grows. Which effects the loading of LB to taking 30+ seconds per edit of a game and at 78 meg.. it's on longer portable anyway. Thoughts: Is portability really needed? yes, the INSTALL is.. but once you adding a few games, the idea of throwing everything to a USB drive and taking it to a friends becomes harder and somewhat impossible. so the question is: What is the why/reason for the need for the portability of the data? #2 - Use a seperate XML file to hold all the meta data Pros: 1) Would be easy(ish) to implement.. perhaps by GamesDb ID (so 8428.xml) which is Postal 3 2) Easy to update specific games (or re-update from the Gamesdb.net) 3) Possible to embed images into the games XML file. 4) Could be used for ISOs, DosBox Games, etc. Cons: 1) easy to de-associate game data from game image. Some sort of clean up system would be needed to sync the XML/image files no longer needed. 2) possible slow down on load due to loading multiple files at once (though, this could be threaded) 3) Makes Search and Filters much harder to implement and would slow the system down having 32k of files to search though in order to produce a search/filter list of games #3 - Embed the meta data of the game WITH with Game itself. Pretty much the same Pros and Cons as above.. except that the association of the meta data and game won't be lost and it would be easier to share this format than to share an image and force the user to download the meta data again. #4 - Hybrid of #2 I think this may be the best way to handle this issue. the PRIMARY meta data (Name, Release Date, Genre, Publisher, rating, etc) would be read from flat XML file as the game is being imported (and the XML file being created).. THis way all searchs and filters are fired agaist the database. If a user wants the additional information about the game.. then the XML file is read into the form. PROS: #1 - Fast access to any of the games, fast search, and filters. #2 - Opens the ability to create a virtual data grid so that a 78+ meg XML file isn't trying to be loaded all at once. #3 - Splits the data needed to RUN a game VS neat 'extras' that some users may not even care about. CONS: #1 - pretty much a total re-write of how the import system works, how the games are displayed, and how the import process works. #2 - Still pretty easy to de-associate the meta data from the game image. So the a tool would still be needed to clean up the XML files.. but now also the database as well. Also a tool to 'compress' the database would be a good idea to keep the database file size down. however if the import is split off.. you can speed up the import process by only downloading the primary meta data and download the 'extra' data later.. in the background.

[SentaiBrad]

pogowolf said Issue: Keeping the meta data downloaded from 'somewhere' (in this case TheGamesDB.net) associated with the game itself. #1 - They way it's done currently is that all the meta data is downloaded and saved in one XML file. (mine is currently 78 meg) which is parsed on the fly for information, possibly partial saved on update, and re-read into LB after each edit. Pros: 1) Keeps the system 'clean' and it easy to access data 2) Keeps the data in once place, making the whole of LB more portable. Con: This process slows as the size of the XML file grows. Which effects the loading of LB to taking 30+ seconds per edit of a game and at 78 meg.. it's on longer portable anyway. Thoughts: Is portability really needed? yes, the INSTALL is.. but once you adding a few games, the idea of throwing everything to a USB drive and taking it to a friends becomes harder and somewhat impossible. so the question is: What is the why/reason for the need for the portability of the data? #2 - Use a seperate XML file to hold all the meta data Pros: 1) Would be easy(ish) to implement.. perhaps by GamesDb ID (so 8428.xml) which is Postal 3 2) Easy to update specific games (or re-update from the Gamesdb.net) 3) Possible to embed images into the games XML file. 4) Could be used for ISOs, DosBox Games, etc. Cons: 1) easy to de-associate game data from game image. Some sort of clean up system would be needed to sync the XML/image files no longer needed. 2) possible slow down on load due to loading multiple files at once (though, this could be threaded) 3) Makes Search and Filters much harder to implement and would slow the system down having 32k of files to search though in order to produce a search/filter list of games #3 - Embed the meta data of the game WITH with Game itself. Pretty much the same Pros and Cons as above.. except that the association of the meta data and game won't be lost and it would be easier to share this format than to share an image and force the user to download the meta data again. #4 - Hybrid of #2 I think this may be the best way to handle this issue. the PRIMARY meta data (Name, Release Date, Genre, Publisher, rating, etc) would be read from flat XML file as the game is being imported (and the XML file being created).. THis way all searchs and filters are fired agaist the database. If a user wants the additional information about the game.. then the XML file is read into the form. PROS: #1 - Fast access to any of the games, fast search, and filters. #2 - Opens the ability to create a virtual data grid so that a 78+ meg XML file isn't trying to be loaded all at once. #3 - Splits the data needed to RUN a game VS neat 'extras' that some users may not even care about. CONS: #1 - pretty much a total re-write of how the import system works, how the games are displayed, and how the import process works. #2 - Still pretty easy to de-associate the meta data from the game image. So the a tool would still be needed to clean up the XML files.. but now also the database as well. Also a tool to 'compress' the database would be a good idea to keep the database file size down. however if the import is split off.. you can speed up the import process by only downloading the primary meta data and download the 'extra' data later.. in the background.

Let me see If i can't answer some of this. 1. Yea, its all kept in one XML file for portability. Portability was the design philosophy Jason wanted to take. Since then we've discussed making a non-portable version of LB. This way we could use the registry and possibly get a voting system done within the program. We've since decided to probably go through the site on that though. Either way, the way its being done now was the way that was picked because the library counts weren't that large. The few of us have gigantic libraries but the average user does not. We've tested 10k items with little to no noticeable lag. The larger the XML file, the longer this can take. We have talked about multiple XML filing or changing the settings system all together, but we haven't decided on anything yet. 2. Most likely, the system of images in a folder is going to stay the way it is now. We could possibly further sort the images by Platform in the images folder but chances are it will stay this way. XML filing is a bit antiquated so adding in multiples and images is not desirable because of the massive lag it will cause. Right now its just looking in the XML for data, but data and images then multiple that by 10k, were back to where we started. The caching of images how ever is going to be improved. He is looking in to ways that will help the RAM issues. The filtering system was also the source of the lag before, but since that was taken care of it now seems that it is because of the sheer amount of data. Even if we change the system, nothing will fix that. At a certain point it is entirely dependent on each system, now that is not to say there can't be even more optimizations because of course there will be, but at a certain size there is nothing we can do. 3. I think its pretty safe to say that Jason does not want to create containers for anything. It will be a major headache for users of any level we think if we start putting rom's and data in to other files. It would essentially be scanning archives for data like a 7zip or rar archive. Even if no compressing is taking place that will cause more resources to be needed to parse that information. Boot times could get out of hand. 4. There is a compromise here, we know this. We just need to make sure we find the best compromise for the way the system is setup now. We want to stress, he wants to make the best possible program but if that means a complete rewrite of systems that needs to be carefully looked at and determined if it will be the best decision for the future. For him to rewrite entire systems would mean also partially rewriting other systems and could potentially take weeks; where as some optimizations can take a few days. Nothing is off the table but it also needs to make sense. Without changing how the system behaves now I believe we can try to get more cores to work for the program, even though I am certain it does use multiple cores right now... could be wrong. Also, the major problem isn't with CPU related tasks, its mostly RAM related tasks. The bigger the library, the bigger your RAM usage will be. I think we're going to look at how image caching works. Instead of retrieving the data at its fullest I believe dropping down the max resolution of images and how many are kept in the active cache will HELP this issue, im not sure it will resolve this issue. Lowering the image sizes will also mean that the max size in LB will probably be lowered. We could also limit how much else is downloaded with the import. Right now its defaulting to downloading all available images on the mass import, maybe changing how that works as options or just limiting it to just the Front and Back covers would help. Either way, it will be a decision on the users part. Also, keep the discussions coming, we do like reading them.

[bd00]

Brad, could you not just have LB create thumbnails on import? Limit the size and only load full size images when viewing the covers. Would that not solve your problem?

[SentaiBrad]

bd00 said Brad, could you not just have LB create thumbnails on import? Limit the size and only load full size images when viewing the covers. Would that not solve your problem?

That could potentially, but I was trying to think of a method that wouldn't require another option. Not to mention I don't think we can scale the images on the fly, that would require them to be vectors and blow the ram usage out of the water. If you limit the size a bit at least you still get them and they can be capped on import. Unless we can come up with a RAM saving method with the slider we have now, but I still think that would require scaling on the fly.

[bd00]

SentaiBrad said

bd00 said Brad, could you not just have LB create thumbnails on import? Limit the size and only load full size images when viewing the covers. Would that not solve your problem?

That could potentially, but I was trying to think of a method that wouldn't require another option. Not to mention I don't think we can scale the images on the fly, that would require them to be vectors and blow the ram usage out of the water. If you limit the size a bit at least you still get them and they can be capped on import. Unless we can come up with a RAM saving method with the slider we have now, but I still think that would require scaling on the fly.

But thegamesdb only has one front cover per game, so if it is too large does LB just ignore it, leaving users without a cover? If not, then there is obviously some resizing going on on-the-fly. No need to convert images, just resize them to a standard resolution, like a max of 500x500 if required but leave the original as is for full screen viewing. I don't see why you need to convert images especially to vector, it doesn't make sense.

[SentaiBrad]

bd00 said

SentaiBrad said

bd00 said Brad, could you not just have LB create thumbnails on import? Limit the size and only load full size images when viewing the covers. Would that not solve your problem?

That could potentially, but I was trying to think of a method that wouldn't require another option. Not to mention I don't think we can scale the images on the fly, that would require them to be vectors and blow the ram usage out of the water. If you limit the size a bit at least you still get them and they can be capped on import. Unless we can come up with a RAM saving method with the slider we have now, but I still think that would require scaling on the fly.

But thegamesdb only has one front cover per game, so if it is too large does LB just ignore it, leaving users without a cover? If not, then there is obviously some resizing going on on-the-fly. No need to convert images, just resize them to a standard resolution, like a max of 500x500 if required but leave the original as is for full screen viewing. I don't see why you need to convert images especially to vector, it doesn't make sense.

Let me try to explain further my thinking then. If we offer Icons, then there needs to be an option because not everyone is going to like Icons, they want their full image. This is fine but pointless, so take out the option. My original thought was to yes, resize the image when the Scrapper nabs the image from TheGamesDB. If we convert the images over to a smaller version, then what you would want is two version of the image, I already have close to 10GB of images with my library. If there were duplicates albeit smaller with a large library, that could get even bigger. So my next thought was to size them on the fly as to help with file size, but that would most likely require something like vectors which then also doesn't solve the size problem. My mind works oddly and I sometimes don't express everything. That said, if we only had the small version of the front and back covers but then kept fan art for Full screen game viewing then this could be achieved with the setup we currently have. Limit the most front facing images then use the fan art on full screen because I doubt a full screen image that has been resized to be smaller would look that good. There would also be the issue of how we do that since there is no standard and my thought is to reduce the image size by 50%-75% on import. It might make imports take longer too but in the long run should be a better option. We could also still show the front and back covers in their reduced form but make sure that the program doesn't expand them past their resolution.

[bd00]

I don't think you should remove the option for full size images period. I like the option to view the covers full screen, especially the back covers and I know other people would too. I understand this would be a perfect solution for you, but it needs to be an option if you take that path because I don't think people will like the "thumbnails or nothing" approach. Vector is no good for images like these, I still don't understand why you would consider it. Resizing images would not affect the import time much. It's pretty simple, quick and light on resources. You could even have it run simultaneously with the downloads. So why the next images are downloading, LB could also be resizing the last.

[bd00]

Also, people with libraries THAT big must have plenty of disk space, at least enough not to worry about an extra small thumbnail front cover.