Open - MAME/MESS filenames for importing - Issue #343

[dingodan]

Hi,

I tried to import the atari 2600 roms from MESS but the scraper detected only 125 of 790. I told the scraper to use MAME metadata (tried without for even worse results). Some of these will not be in the DB - not worried about them. But some are in the DB and are not matching (river raid, for example). Ive attached the xml that comes with mame 180 for this system and a list of my file names. These match the XML and clrmamepro confirms this is a good set for 180. 

Is mess filename scraping something that should work at the moment? If not, any advice on how I can get the title-filename mappings from the mame xmls into the games db?

Edit: I can, and will, extract all the roms to get better matches. I just wonder if we could also get matches for the short archive names (id rather leave them compressed with original file name - easier for set updating)

a2600.xml

a2600.txt

Edited December 7, 2016 by dingodan

[DOS76]

LB only recognizes the short names for MAME's arcade side of things. I'm hoping this will change in the future it has been mentioned a few times but most users don't use the software lists for their console games so priority isn't that high

[dingodan]

Ah well, so long as its on the list - thank you for a prompt reply!

[SentaiBrad]

You can just use a regular rom set with MESS, or in the Atari 2600's case with RetroArch. I have made a few tutorials where MESS was the only option, or the better option, and I used games with regular names.

[DOS76]

MAME lets you use regular names also for Software List stuff

 

[dingodan]

Maybe I misunderstand, but wouldn't both of these suggestions require that I have a differently named romset? 

[DOS76]

Yes that or you will have to change the names individually. Unfortunately at this time without support for the software lists you don't have any other option for LB.

[dingodan]

Bummer - I'm really impressed by the softwarelist concept. TOSEC too. Very ambitious and I hope they both succeed. I recognise that I'm not in the majority, but still look forward to support for their naming conventions in future versions LB. 

Is there a way for the community to contribute to file name->title mapping for the LB scraper? Not directly via source, just a form or a google doc where we could provide file-title maps to help improve the scraper over time? I could also make a bitbucket proposal and attach the XMLs, if you think that's appropriate. 

My workaround, for what it's worth:

In case anyone else out there has a merged 2600 SoftwareList or TOSEC set, here are some bash scripts to make them more friendly for the LB scraper.  When importing it is important to not select MAME or use MAME metadata. In Windows, GitBash does the job just fine. 

First SoftwareList; this will extract the first bin file found in each 7z archive, to a destination dir of your choice:

#/bin/bash

destDir="../Atari/2600"

files=`find ./Roms/a2600 -name *.7z`

for f in $files

do

	first="`7z l $f | grep '.bin' | grep -oP '^.+\.\.\.\.\. +\d+ +\d*+ *+\K.+' | head -1`"

	cmd='7z x -o"'$destDir'" '$f' "'$first'"'

	eval $cmd

done

This should get you 729 files. They are all named lowercase - I couldn't be bothered renaming for Upper Case Words. 

For a merged TOSEC set the filenames are more conducive to a successful LB scrape and they Respect Title Case, but there are many different versions of each game. To cut them down to *almost* one of each I used this (reduces 1800 to about 500):

#/bin/bash

sourceDir="./2600"

destDir="../output"

cd $sourceDir

i=0

while read line

do

    files[ $i ]="$line"        

    (( i++ ))

done < <(ls | grep -v 'ZZZ\|Contest\|Cassette' | grep -Ev ' v[0-9]')

m=0

for f in "${files[@]}"

do

	if [[ $f == *"("* ]]

	then

		var=`echo $f | grep -oP '.+ \('`

		var=${var::-2}

		match=0

		for item in "${processed[@]}"

			do

				if [[ "$var" == "$item" ]]; then

				match=1

				fi

			done

		if [[ $match == 0 ]]; then

			processed[$m]=$var

			x=$(printf '%q' "$f")

			cmd='cp '$x' '$destDir

			eval $cmd

			echo $f

			(( m++ ))

		fi

	fi

done

Neither of these are perfect solutions, but they are certainly better than manually renaming. Hope somebody finds them useful. 

Edited December 8, 2016 by dingodan
linefeeds in code

[SentaiBrad]

I think the No-Intro sets are better than the software lists, but it's just a difference. for CD-i I got the CHD's to work over bin and cue, so it's situational. TOSEC has too much other stuff bundled, but at least it has proper names. Some software lists do to, like CD-i, it had the full names. As long as it has the full names you are in business, but if it has MAME styled short hand, you'll need to convert it or LaunchBox will, and it only converts mame currently.

[dingodan]

This might seem dumb, but I couldn't let it go. I think I've actually come up with a cool workaround that can help others: use the XMLs included with MAME to read the full name of each rom for each system, then make symbolic links of the full name. This way I can have my MESS set, but represent it in a friendly way - even with UTF8 support. Then I import the symbolic links from Launchbox and my unsupported romset can be used without renaming it (which breaks other apps that also use the same set). 

PowerShell this time, US roms preferred: http://pastebin.com/D2mxhw3Z

Edited December 12, 2016 by dingodan

[Vlansix]

Hi all,

MAME compatibility / updates has been requested a bit. I'll be looking into this once I've ironed out the bugs in the DB. Check the tracker for updates!