shouting loudly

Netflix: $1m to improve our collaborative filtering

Netflix has a software program that helps you find movies you like, much like Amazon’s system helps you find stuff you like in general. CNet explains:

Recommendation systems, also known as collaborative filtering systems, try to predict whether a customer will like a movie, book or piece of music by comparing his or her past preferences to those of other people with similar tastes. Such systems will look at, say, the last 10 books, movies or songs a customer has rated highly and try to extrapolate an 11th.

But they’ve hit a kind of wall. If you can improve the current system by 10%, you win a cool million.

There’s another twist to the story: to make the contest work, they have to release the database of rental histories. Unlike the AOL search data debacle, however, Netflix carefully considered the privacy implications and got the nod from privacy experts. The data are also just easier to anonymize; a person’s web portal search records are generally much more personal than a list of rented movies.

Researchers are jazzed to finally have a big data set to work on. Having watched my colleagues and professors jealously guard the large survey datasets floating around Annenberg (those who get more credit for assembling and managing the survey, weighted by seniority, informally get to decide which lines of research are theirs), I can imagine the excitement of university researchers and other engineers outside the labs of major for-profit web companies.

So fire up your terminals; the contest will be officially announced today.

(Here’s the same story at the NY Times.)