How The Mightily Hyped Have Fallen

Second Life inspired a wave of hype and fear not seen since... well ok we see that kind of thing all the time, but it was like Internet Hype 2.0, which of course was another version of Digital Utopia (version number too large to count really). So, every company had to get into Second Life, even when much of it turned out to be weird and porny. They got in like there was no tomorrow, massive announcements, and then quietly left, trying to hide their embarrassment. Even Wired left their offices.

I just ran into this ad on a wiki. Wow. They've really hit bottom. Even I am a little embarrassed for them. (Google will never do text flow correctly, drives me nuts.) "YES WE ARE SLUTASTIC." Nicely done, Lindens (if it is even still the Lindens), embrace your scumbaggery.

Silver's "The Signal and the Noise"

There are some errors in Silver's (excellent? interesting? occasionally vexing?) book from a little over a year ago, The Signal and the Noise, about prediction and Bayesian theory. The two I noticed are strange, because they are very basic. (Other people have noticed these errors.)

On page 269, the book says how 20 x 20 is 4,000, when it is actually 400. This is pretty basic. Would Silver make this kind of error? Was he rushed? Did he and the editors miss it? Did someone else ghost write parts of the book? If this error is here, what are the errors that I didn't notice?

There is also an error or two with the arrows-length illusion image on page 367. For one, the description is not clear, and it seems that Silver is describing the illusion in the wrong way (that is, he says the one that appears to be longer appears to be shorter). However I measured the lines of the arrows in my printing, and indeed the one which is supposed to appear longer via the illusion is indeed longer by a millimeter. It looks like a printing error, where the lines may have been the same length, but when the arrow heads were added, they were added compactly to the "will look shorter" line but were not to the "will look longer line". The thickness of the added arrow heads is just enough to throw the whole illustration off. So not only does Silver's description get it wrong, but then the illusion isn't an illusion, one is actually longer than the other.

This really undermines the entire book, since I'm reading it to learn about things I don't know, and if Silver can't get right the things I do know then I know I have no idea if he is getting right all of the material I don't know.

Unrelated to these errors, I did like this review: http://mathbabe.org/2012/12/20/nate-silver-confuses-cause-and-effect-ends-up-defending-corruption/

Counting What Counts

(…and the qualitative versus quantitative divide.)

Not everything that can be counted counts, and not everything that counts can be counted.

-William B. Cameron, sociologist, 1967.

℅ Anthony Towndsend's book Smart Cities.

William Whyte and Benches

One thing I recall from William Whyte's fantastic The Social Life of Small Urban Spaces is one idea about people: people like other people (so we like people watching), but some people (typically who own, maintain, or work next to a small urban space) don't want "undesirables" in that space while "undesirables" don't want other people around and so you want things like benches.

Here in NYC, we have some recent citywide benches. People can sit, relax for a moment, and watch other people. Passersby might feel a little better with some other people sitting there. But, what's interesting about these benches is the little raised seat dividers -- people can't sleep on these benches, probably addressing the "undesirables" issue that Whyte wrote about, while the rest of the bench lets people sit there, as we like to do.

Halloween in EQII

Well sure, there's Nights of the Dead or whatever, but here's some fun homage I mean IP infringement almost I mean homage. Hopefully the text flow is ok, but we have "Norm Baites" and we can also wonder how many licks it takes to get to the center of a… well just a lollipop.

R and Regex Named Matches

I use Python and R to do stuff, Python for web scraping and text clean up, R for the analysis. But people have expanded the functionality of the two, and they are overlapping (it's enough to not get them confused as it is some of the time). I found I needed to use named groups in regex in R, and... couldn't figure it out. The web did not help.

SHORT VERSION: Turn on the Perl regex style (perl = TRUE) and go read some Perl regex pages, you'll be fine. Name the match: (?<name>...), to match it later: \\g{name}
This is completely different from what I was used to.
Google blogger will try to blow this post up as I want to have greater than and less than symbols. Yeah they can't get that right. Oh maybe it's working.

Typically, if I use regular expressions it's in Python, but R can do it too, and sometimes you'll want to do that. But there isn't a ton of help online about it (despite the links I have lined up to include below) and there are some things that confuse the issue (to Perl or not to Perl...).

If you just want to do some work with strings, first check out Hadley Wickham's stringr. It's awesome.

I, however, wanted to do some pattern matching that included a repeated section, so I needed regex's named group functionality, which I couldn't find or figure out in stringr. I was looking for patterns like this:

5,-1,5,-1,5

...where 5 could be any number between 0 - 500 or so, but it would repeat. I had already removed spaces and added commas for easier parsing. (So other matches would be, like, 17,-1,17,-1,17.... etc.) So I needed to make sure the first match there was repeated, thus, named groups (or any group capture really, but I wanted to name it).

But I also couldn't figure it out in R. I can do it in Python, but the Python code for regex wouldn't work in R, alas. It was not clear what changes needed to be made.

One reason was the the \ needs to be escaped, that is, \\. So for example, \d+ needed to be \\d+. That wasn't too hard to figure out. But the rest was.

You can have Perl style, or POSIX, or not. Uh, what? No idea! I just needed it to work. Specifically, named groups in R. I found this page which said "Named subpatterns... are not covered here." Hmm. Another page said how "examples for the use of regex in R are rather rare" and had some useful examples. Eventually I figured I would set the Perl option and see what I could do; at least I could search on "perl", and that made all the difference as I could find out how to do named groups in Perl-style regex and there you go.

Name a group in Python: (?P<name>...)    
Name a group in R, Perl style: (?<name>...)

Note: I expect the less than and greater than symbols fail at some point.

Reference it later in Python: (?P=name)
Reference it later in R, Perl style: \\g{name}
    So curly braces (Perl?), and double backslash for R.

Some useful Perl-regex links:
http://modernperlbooks.com/books/modern_perl/chapter_06.html
http://perldoc.perl.org/perlre.html

Although honestly one problem I have with a lot of online examples (and the R help files) is that they are completely arcane. If I'm looking for help with syntax, a complex example isn't going to solve it, that's bad usability.


Post Keywords: regex, R, r-project, cran, grep, regular expressions, named groups.

iTunes Radio

Still working out a few kinks, as you can see (not repeating recently played songs). The result is I have had Fiona Apple stuck in my head for four days. (Not 'shopped.)