Wednesday, August 12, 2009

Competitive Crowdsourcing

The Netflix Prize reached its final stage last month almost three years after it began. For those unfamiliar with the competition, Netflix posed a challenge to the scientific community to develop an improved collaborative filter algorithm for use in determining Netflix customers’ cinematic tastes. The challenge was to improve on the current Netflix algorithm by at least 10%. Although a small improvement in percentage terms, the target was much harder than originally thought and has attracted 51,000 researchers since 2006. Ultimately there were two winners here. First is the research group that claimed the $1m prize. And then there is Netflix itself. The competition was a clever idea, and proved much more capital efficient then hiring scores of engineers, outsourcing or buying a collaborative filtering start-up. With minimal upfront costs Netflix now has a much improved product that is worth considerably more than the $1m payout.

Aside from learning about the fascinating world of collaborative filtering and algorithms, the exercise revealed the competitive nature of scientists, mathematicians and engineers. Of course, the prize money was an important catalyst and attention grabber, but it was the prestige associated with the battle of intellectual minds that lit a spark and kept these math geniuses going although way to the final stage.

Pride has always been a strong driver of innovation in the scientific and academic community, but is distinct from the pride of an entrepreneur, who is foremost driven by economic considerations. In science, there are grants and there are prizes. The former funds specified projects in the hope they succeed (e.g. NIH, DARPA). The latter are awarded in a fairly subjective manner for past achievements (e.g. Nobel, UNESCO, etc.). What we clearly don’t see enough of, are competitions, where prizes that are awarded for meeting a defined and measurable goal.

It is perhaps less scientific and more commercial, but such competitions can galvanize both entrepreneurs and scientists to pursue a worthy goal…and I am wondering why we don’t see more of them. After all, competitions not too dissimilar to this once spawned the world’s first marine chronometer and parking meter. Such competitions are a form of “crowd sourcing,” except that there is a fairly specific goal and a payout to the winner.

It reminds me of the X-Prize Foundation, which promotes scientific and entrepreneurial achievements that benefit humanity through $10m awards to entrepreneurs and scientists who are the first to achieve an objective goal. The most famous of these awards was the Ansari X-Prize, which awarded the prize to the first group “to launch a spacecraft capable of carrying three people to 100 kilometers above the earth's surface, twice within two weeks.” In retrospect, the monetary reward has little to do with spurring teams to win Ansari X-Priz, as the prize money is almost certainly not the incentive given the vast sums of money often invested in the projects.

Often forgotten is the fact that the Ansari X-Prize was modeled after similar prizes given away in the early twentieth century to spur the creation of the aviation industry. Most notably, Charles Lindbergh won the Orteig Prize for flying across the Atlantic in fixed wing aircraft. Here the cost was most certainly more than the $25k prize, as 6 lives were lost in 3 different crashes.

The recent success of the Ansari prize in turn prompted Richard Branson and Al Gore set up the Virgin Earth Challenge, which promises $25m to promote technologies that removal of greenhouse gases. Cisco too has the I-Prize. However, the Virgin and Cisco competitions are more akin to awards for past achievements, as they are not well defined, objective competition.

The more nagging question I have is why more companies and government institutions don’t use a similar strategy for solving complex problems. Perhaps they are simply afraid to reveal their technology holes or give away ideas to entrepreneurs. Nevertheless, for corporations brave enough, companies such as Innocentive, BrightIdea and Big Carrot offer a hosted platform, or exchange, for creating competitions and sourcing ideas/solutions from the masses. I look forward to seeing companies, large and small, use competitive crowd sourcing in the future, as it may prove a core part of corporate R&D, just like M&A and outsourcing are today.

And what a case of serendipity that as I write, I discover that Netflix is preparing a Netflix Prize 2!

No comments:

Post a Comment