Open innovation in collaborative filtering

Netflix has just announced a $1 million prize to whoever can improve the accuracy of their movie recommendation engine. To enable people to design an improved recommendation engine, they’ve provided their users’ ratings of 100 million movies, an extremely valuable database. This harkens back to Canadian gold mining company Goldcorp’s initiative, whereby they publicly released the geological data on their properties, and set up a competition with prizes for whoever could give them the best recommendations on where to dig for gold. Other open innovation initiatives such as Innocentive match a whole series of people looking for innovation, again providing pre-specified rewards for meeting specific parameters. Some note that the prize will mean a lot of people work for free, and it's arguable that if you can indeed do better than the other competitors, you'll be able to make more than $1 million from it commercially anyway. The size of the prize indicates the value in enhancing the accuracy of collaborative filtering, as I’ve written about many times before. If Netflix can more accurately recommend a movie to its customers, the more likely they will stay with Netflix. For companies with other business models, greater accuracy directly impacts sales and revenue. More and more energy and resources will be going into this space. Netflix has chosen to combine two of my passions – open innovation and collaborative filtering – so I will be very interested to see the results from this. Details of the prize are at netflixprize.com, which will provide a progress chart on how the competing teams are doing.