Big Data Divide

About a month ago I was at CeDEM, the Conference for E-Democracy and Open Government, in Krems, Austria. Several projects were focused on communication between local municipalities and the citizenry, with a focus on communication and knowledge sharing that could go either way (such as, a municipality learning what local citizens don't know, and getting that information to them). So a lot of it concerned data collection and analysis.

That should sound familiar, given recent events. I pointed out how there were two sectors that already had the three needed factors:

1. Data collection capability,
2. Data (have been collecting),
3. Data analysis capability.

These two sectors are, of course, intelligence agencies for at least the US and perhaps Britain (and other governments on a smaller scale, most likely), as well as the information companies themselves (in this case I mean information conduit providers, like Google, AT&T, and Verizon, to name a few).

Sure we could pass laws forcing these two sectors to help out local governments provide better services (one eternal promise of not just the internet, but of the information future -- it's always a day away), but I doubt that will happen. That's rather sad, and falls short of the "of the people, by the people, for the people" ideal from US President Lincoln.

The NSA and "Unreasonable"

The 4th Amendment to the US Constitution, part of the Bill of Rights (the first ten amendments, as a package, minus two), ratified in 1791.

Perhaps the, or a, pivotal section relies upon the word "unreasonable":

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated...

Is it unreasonable to have the government have access to data that the giant info-companies already have access to and have in fact generated and searched through themselves? It certainly seems unparalleled, and I don't for the life of me see how it is actually legal, but given that other organizations have been doing this for years for their own good I have a bit of a hard time seeing how it is unreasonable to have some other people do the exactly the same thing as others while looking for terrorists instead of for their own capital gains.

People, especially at Wired Magazine, used to love to name-drop Bentham's panopticon back in the 1990s, but that idea hasn't been used much if at all in the recent dust up. Perhaps it seemed cool until it happened.

Edit: Last week's New Yorker lead article does mention Bentham, but since I was off at ICA in London I only saw it today. The article also astutely points out how we knew the NSA was doing this for years, and knew this years ago.

More Game Homage

Two more interesting bits of homage from some games:

• EverQuest II and Back to the Future
• Minecraft and Portal

Most of the homage in EQII is fantasy related, but on occasion there is general nerd humor (like at least one for Superman and one for Star Trek, although there is also one from The Loveboat, which is unusual). So, sci-fi in fantasy, even fantasy like EQII and WoW which have steampunk-like gnomish tinkerers (when did this happen to allow these games to do this?), is unusual.

Back to the Future's flux capacitor in EQII:
(EQII link) (BttF link)

As for why it's a hat, I don't know, but there it is.

Everyone, even my 9 year old nephew, knows and loves Minecraft. Everyone should also love Portal, since it was amazing (as is Portal 2). But, Minecraft is strongly influenced by some other games (as are many things influenced by other things), among them one of my favorites Dwarf Fortress. Portal was not among them as far as I know, but here you have the infamous cake, which, as it turns out, really is a lie. (Or is it, yes, yes it is, but you can think it's as delicious and moist [at 2:18] as you like.)

The cake is no lie in Minecraft, except it is:

(Minecraft link) (Portal link)

Copying helps us make things.

Song and Dance

Given the connection between community and communication (as can be easily seen in the English words, from Latin), I somehow was clued into the importance of music and dance as community reinforcing activities. Communities are, in some ways, where a group of people coordinate their actions with each other -- the same is true of dance (and the music that goes along with it). So I had an "of course!" moment when I read this:

The Sesotho verb for singing (ho bina), as in may of the world's languages, also means to dance...

From Daniel Levitin's This Is Your Brain On Music, p. 7. I'm not a linguistics expert, and I know English (sing, dance) and Spanish (cantar, bailar), and as you can see neither of those clued me in.

Online Dating Math

If 1 in 5 (20%) relationships start online, that suggests that 4 in 5 (80%) start offline. Get away from your screen and go out into the world! Or something. And "See Pics of Single Women", really? What is this, a porn ad? I don't see that the Match.com people could have thought about the math much, or they think their audience is not very good at math. And yes they are probably trying to highlight the word "Free" but it shouldn't be capitalized, and they should know this is the internet where there are people who can do both math and grammar and insist on them being done correctly.

R and Date Formats

There are many ways to represent dates in computer languages. You can have something human-readable, with a year, month, and day, in some order and some combination of text and numbers.

• 2012-11-19, year month day.
• 11/19/12, US month day year.
• 11/12/11, Nov 12 US or Dec 11 EU.
• Nov-19-2012, a combination of text and numbers, to a human.
• "2012-11-19", which is probably text to a computer given the quotation marks, but numbers to a human.
• 1353340800, a big number that means nothing to most people, but is today at 4pm in seconds since Jan 1, 1970. This is a standard time format, I've found.

We will gloss over leap years, leap seconds, and different historical calendar systems, but it's all a fascinating and nuanced topic.

But I ran into a huge (yet small) problem with time formats in R, and trying to convert from seconds since 1-1-1970 to some current dates, where I had 45,000 data points. If I tried to analyze them, they ended up as 45,000 different points and didn't aggregate at all. Hard to visualize. So I wanted to aggregate them all by month, and instead of 45,000 points I would have, say 120 (10 years, 12 months each). I'm taking averages, and this approach makes sense. This turned out to be harder than I thought -- essentially I was just trimming everything but the year and month (days, hours, minutes, seconds). Simple!

Edit: I'm including a screen shot of the code, so Google doesn't destroy it (due to the characters in it and how the R code is interpreted by the Google HTML parser).
"the_df" is the data frame.
"cm" is creation month.
"bday" is the seconds since 1-1-1970.

The "as POSIX" call converts the number of seconds as a number to number of seconds as time.
The "strftime" strips off the day (off the end) and makes it a text string (string-from-time, I think).
The paste is the key part, it won't work without it, you have to add the day back on (the 1st of the month for this analysis). I think this makes it a string value (not a date).
Then the "as date" call converts it to a date from a text string.
(Or something generally like that.)

The point is, if you use "as.Date" in R, you have to hand it a day. I tried just handing it year-month, but that ends up as NA, even though I can't find anything that says that kind of thing shouldn't work.

Election Numbers and Massachusetts

(I could reframe this in the following manner: The most telling vote of the election... was it how women voted? How minorities voted? Latinos and Hispanics? Regions? White people? Evangelicals? No. The most telling vote was instead from one small state in the corner of America: Massachusetts.)

Perhaps the most telling numbers from the election come from Massachusetts--the only state to have direct experience with both President Obama and Mitt Romney as chief executives. Massachusetts voters had four years of Mitt Romney as governor, and also had four years of President Obama as US President. Given this unparalleled experience, how did they vote?

The results are clear.

Massachusetts

Obama	1,900,575	60.8%
Romney	1,177,370	37.6%

That's a one-sided endorsement. (Percentages do not add to 100% because I didn't include the "Other" votes which are part of the total.)