Since the month ended…12 days ago I’m calling it quits on the Netflix Prize. It just wasn’t in my wheelhouse. I was playing around with a few things, but tying to come up with a decent algo using command line PHP on my AMD XP 2200…it just wasn’t happening. I may still play around with it if I stumble upon some brilliant idea, but for now I’m moving on to a new project. Official announcement to come.

Popularity: 6% [?]

I’m starting to realize that the key to getting a decent RMSE is to not wiff completely. What I mean is you want to minimize the number of times your predicted rating is 2 or more away from the actual rating. So the big problem really is determining when people are going to rate a movie 1 or 5 stars. I’m going to try not to worry about 2,3, or 4 star ratings. In those cases I could predict a 3 and at worst be 1 off. It’s when someone goes to either extreme that it can really hurt your score.

Take user 1858615 for example. User 1858615 has rated 227 movies, giving a whopping 183 of them 5 stars (80%). But they’ve also dished out 27 one star ratings (12%). That means that 92% of their ratings are either a 1 or a 5. This user has 5 movies in the probe set, which I have to predict thier rating on. Of course the user’s actual ratings are 1,5,5,5,5. Guessing the 5s isn’t so hard since the user has a penchant for high ratings, but how do I sniff out that 1 star rating? The movie in question is “Kung Fu Hustle“, which averaged 3.73 stars. Not exactly a dud.

My challenge now is to dig deaper into the data and try to figure out why user 1858615 gave movie 10231 one star.

Popularity: 4% [?]

Just for I decided to see what RMSE would be if I just assigned each rating prediction a static value starting at 1 and increasing by .5 until I hit 5. Results below:

Popularity: 4% [?]

I finally have all the data loaded and can begin messing around with some algorithms. The table of movie ratings is over 100 million records long. Even with indexes, it just too big to do any sort of grouping or joins on. This rules out algorithms like “User A has rating x number of movies very similar to User B. Therefor they will probably feel the same way about movie y.”

I can’t imagine this algorithm would be of any use to Netflix anyway.

They have 5.6 million subscribers currently and predict they’ll have 6.3 million by the end of the year. Plus they claim to have over 65,000 titles. That’s somewhere around 364 billion ratings/estimated ratings.

For this thing to be of any use to them it’s going to have to be pretty darn fast.

I threw together some simple tests just to get started. First I took the average rating for each movie and used that for all estimated ratings. This yeilded a RMSE of 1.0519. Next I took the average rating for each user and used that as the estimate. This yeilded a RMSE of 1.0426. This is a bit surprising. How the user voted on previous movies is a better predictor to how they will vote on future movies then how others voted.

More to come…

Popularity: 4% [?]

I just got this e-mail from The Netflix Prize:

After discussion among the judges we have decided to reduce the time between prediction submissions from one week to one day. This change will go into effect this Thursday, Oct 19, 2006, 16:00:00 UTC. A new copy of the Rules and the main FAQ will be posted at that time, reflecting this change, a clarification on the constitution of teams, and other helpful suggestions from many of you. We'll be updating the Forum FAQ entries to match soon after the change.

This is HUGE…you know assuming I ever the time to actually make an entry.

Popularity: 4% [?]

Some random factiods from the data provided by Netflix. Netflix uses a 1-5 rating scale, with 5 being best.

Top 5 movies, by average rating:
The Lord of the Rings: The Fellowship of the Ring: Extended Edition, 4.72
Lord of the Rings: The Return of the King: Extended Edition, 4.72
Lord of the Rings: The Two Towers: Extended Edition, 4.70
Lost: Season 1, 4.67
Battlestar Galactica: Season 1, 4.64

Bottom 5 movies, by average rating:
Avia Vampire Hunter, 1.29
Zodiac Killer, 1.35
Alone in a Haunted House, 1.38
Absolution, 1.40
Vampire Assassins, 1.40

Top 5 movies, by total votes:
Miss Congeniality, 232944
Independence Day, 216596
The Patriot, 200832
The Day After Tomorrow, 196397
Pirates of the Caribbean: The Curse of the Black Pearl, 193941

Top 5 hate it/love it movies (25% of votes are either 1 star or 5 stars)
Dragon Ball Z: Trunks Saga, 622, 761
Dragon Ball Z: Vol. 17: Super Saiyan, 365, 429
N Sync: Live at Madison Square Garden, 247, 183
Battle Athletes Victory: Vol. 7: The Last Dance, 101, 98
Pokemon Master Quest: Collectors Box: Quest 1, 55, 59

Popularity: 4% [?]

I’ve decided to mess around with the Netflix prize for the rest of October. At the same time I’ll be trying to get the word out about FantasyLife, RSStalker and PlanbookEDU. Planbook is a site I built for my wife, a third grade teacher, a few years ago. Although it isn’t a SiteAMonth site, it probably has the most active users of any of my sites and has definatly brought in the most $$$.

The Netflix Prize is a contest to improve there rating prediction system. Based on a customers rating of movies you have to predict how they rated a certain movie. I have a few ideas, incorporating some of the stock market / sports prediction systems I’ve done in the past. Although I can’t even get started because it’s taking me a solid day to load the data. The files are 2 gigs uncompressed and over 100 million rows once formated for MySQL.

I’ll probably post the code I used to format and load the data once I have it actually done. I’m also interested in checking out some random stats, like worst rated movie, so I may post about that too…

Popularity: 4% [?]