Artist countries

I've long wanted to add artist country of origin data to lol.fm. Here are some notes on how I might eventually go about it.

Getting data

MusicBrainz seems like the best bet still. I downloaded a JSON set here containing all artists.

I ran

tar xzvf artist.tar.xz

This gave a ~16 gigabyte JSON file called artist.

Getting countries

I isolated artists and countries from the set, separated by tabs. Artists can share a name, so I also needed some kind of proxy for renown. I didn't want to match "Genesis" against the act from Uruguay for example. I used .rating.votes-count as this popularity indicator; I probably want to match against the most voted-on picks.

cat artist  | jq -r '"\(.name)\t\(.country)\t\(.rating."votes-count")"' > countries-voted

This resulted in a ~54 megabyte plaintext file.

Finding countries

My input was a newline-delimited plaintext file of every artist logged in my library, called my-artists. I wanted the first sweep to be dumb, posixy, and as inclusive as possible, so I ran the following.

tr '\n' '\0' < my-artists | xargs -0 -n 1 -P 8 -I {} sh -c 'grep -Fi "$1" countries-voted | awk -F"\t" -v search="$1" "tolower(\$1) == tolower(search)"' _ {} | grep -v null > matches

I hate it.

  1. Artists can have all sorts of weird names with strange quoting rules, so I ran tr '\n' '\0' to end each line of input will a NULL delimiter, and then xargs -0 to only delimit by NULLs.
  2. xargs also parallelized its inputs with -n 1 -P 8. The latter argument can be scaled according to your number of CPUs.
  3. grep -Fi should only match raw case-insensitive strings, no regex; dunno if this makes things faster.
  4. Everything else is about filtering out substring matches, dodging all sorts of quote-escaping nonsense. If I didn't do this, I'd match the US act "Secret Agent 23 Skidoo" for the British "23 Skidoo".

This is slow and can almost certainly be done better, but it can almost be understood at first glance.

I sorted by names and popularity.

sort -t$'\t' -k1,1 -k3,3nr matches > sorted-matches

Now take the first result for each artist.

tr '\n' '\0' < my-artists | xargs -0 -n 1 -P 8 -I {} grep -Fi -m 1 "{}" sorted-matches | cut -f 1,2 > artists-to-countries

Accuracy

In terms of raw artists, the hit rate is pretty bad.

$ wc -l my-artists
    4099 my-artists
$ wc -l artists-to-countries
    2670 artists-to-countries

And some of that is almost certainly the wrong country.

But when we consider the long tail of weirdly-spelled compilation fodder that bulks up this number, it's not so bad.

$ cut -f 1 artists-to-countries | sort > matched-set
$ grep -vFf matched-set my-artists > diff    
$ sort -R diff | head -n 30
Scidoo's Dead Slang
und Piloten
Interface Crew
Überdruck
Caique
Pro Lagwee
Orchestre Super Jheevs des Paillotes
Panicsmile
Jack Novin
Usward
Ｓｔｉｌｌ
Badia
Version
mrSimon
Quinn Martin
Pegmo
Walter Schmidt
G-Schmitt
Ken Snyder
No Critics
Univers Zéro
Photodrama
POWERPLANT
Tobi Or Not Tobi
Freiwillige Selbstkontrolle
Lolita Psychodelia
Fantastic Plastic Machine
CrapHazzard
De'Lacy
Zumen Featuring Leafnuts And Aaron Phiri

I still have to give the exact modelling a think, as well as make it easier to update country info manually, but this is a decent start.