Hello all,

I am hoping to play around with some data, just to become more familiar with different data analysis techniques, but I am not familiar enough with how to program something to extract the relevant data from List Juggler.

The about section says that list juggler is a combination of python (server side), MYSQL (database), and javascript (client side). Does this information affect what language I would use to extract all of the data, or could I use my language of choice?

Does any one who is familiar with this sort of thing have a language they prefer to using to crawl through a web page and extract data? A pointer about where I could go to read up on the techniques to do this would be great.

In the end what I would like is to extract a list into a high dimensional vector for each list. It would have an element that is between 1 and 0, and has the proportion of games won in swiss, another showing the MOV, possible one having the strength of schedule, and then another field showing how far (if at all) the list made it in the top cut. Next there would be an ID number for each tournament, so that I could access what other lists this may have had to face. Also, I would probably want to include something about the date, so I can look at the passage of time. Finally it would be followed by fields related to cards, where there is a spot for each pilot and upgrade card. Most of these would be 0s because the item is not in the list, and the rest would have the number of copies in the list.

The problem is that this representation does not encode which ship had which upgrades, but I think it would be a good first pass at how to analyze the data.

I would be interested in doing some regressions relating each upgrade to MOV or to winning percentage (or both).

Thanks for any tips about accessing this data. I have some programming skills (Matlab and Java, and to a lesser extend Python, C/C++, Scheme/Lisp), but have not worked on the kind of code to crawl the web and extract information.

Hello! I'll send you a PM to walk you through how to get the data.

Short story for anyone else observing: you don't need to crawl the page - I check the database into source control once an hour, so you can just pull the db down and shove it into mysql.

EDIT: there's also a static http link if you want to download the db via url

EDIT: in github there's a schema diagram too

This sozin fella is an alright dude.

Thanks! I have the database, and have access to it. I still don't understand all of the relationships, but I play looking at the data and the diagram I should be able to start figuring things out.