Thursday, October 22, 2009

Crowdsourcing Confusion

Crowdsourcing is usually used in either one of two incompatible ways, or in a way that is an odd mishmash of both. One the one hand, it can mean throwing some problem out to the crowd, that is, everybody, and everybody solves it. The second is that you throw a problem out there, and the experts solve it. Experts are not everybody, and everybody, as a whole, is not expert at everything. Most people are experts at perhaps one thing, or less.

The recent Netflix improvement competition, to improve movie recommendations, is one such example. One Wired writer said how "the Netflix Prize competition has proffered hard proof of a basic crowdsourcing concept." The NYTimes Bits blog writer, Steve Lohr, wrote "this kind of Internet-enabled approach, known as crowdsourcing, can be applied to complex scientific and business challenges."

Well, no. The winners of the Neflix prize were not at all the same "crowd" as, say, the Wikipedians (and given that the majority of edits come from a minority of users there, that's not really a crowd either).

The point is that you throw a problem out and hope the get the experts interested in it, you don't care about the crowd one whit. The problem is you don't know where the experts are, and you don't have the capability to approach them (perhaps you lack the time, or the social capital to talk to them directly).

The Netflix prize instead shows the well-known point that teams with diverse backgrounds can come up with better answers than groups where all the people have the same background. As Lohr wrote, the winning team "is a seven-person team of statisticians, machine-learning experts and computer engineers from the United States, Austria, Canada and Israel." Not all computer scientists. Not all statisticians. Not all from the same country. If all you have is a hammer, you'd better hope you only encounter nails. (Van Buskirk got this correct in Wired, "Arguably, the Netflix Prize’s most convincing lesson is that a disparity of approaches drawn from a diverse crowd is more effective than a smaller number of more powerful techniques.")

Even the TV show House played with this idea recently. An annoyingly Internety guy blogs his medical problems and offers a reward for them. The person who solves his problem, it turns out, is not just some guy from "the crowd". It is a medical expert. Not just any medical expert, but Dr. House himself, the uber-expert.