Verified by 
A few years ago, Netflix offered $1 million to anyone who could solve the following problem: Given a list of movies someone likes, successfully predict other movies he or she will like. Mathematically, the solution appeared very simple, using a process known as "regression," which is as easy for stats geeks as adding two and two.
But it failed. Even though, technically, a group did win the prize, Netflix doesn't make much use of the inning winning algorithms. Such prediction models have gotten incrementally smarter since then, but no major leaps have been made. Standard approaches continue to be used, just with some tweaks.
Why are even the best algorithms still so bad are predicting what we will like, feel or do? The answer has to do with the complexity of the human preference system.
Challenges like Netflix's, it turns out, make some very inaccurate assumptions about the way preferences work.
First, they assume that our preference function is simple, when we are multidimensional creatures. If we created a multidimensional space of movies and gave each movie a point in that space, our space would not be a cloud, but a lumpy thing with lobes and branches. Some people like westerns and sci-fi, but not martial arts movies, for example, even if an algorithm thinks they should. More often, to make things more complicated, people like certain westerns but not other ones, just as they like certain scenes within a single movie more than others. And how much we like a movie cannot be determined by just averaging how much we like its parts. It's a more complicated non-linear function.
But things are even more complicated than that, as expressed by the mathematical approach called "second-order complexity," or the idea that sometimes, we like things that are different. You want to see superhero movies one month, but then you want to see something totally different for a while. Or you see "Iron Man," even though you don't generally like superhero films, and that gets you interested in "Iron Man 2," and in other movies with the same actors as that film. You see too many war documentaries, and then you want to see a rom-com. Your preference on any given day is influenced, then, not just by what you saw recently, but by what you haven't seen recently. And your likelihood of wanting a big jump to see any given movie is really hard to guess.
But it’s actually even more complicated than that: Our preferences are not static. We oscillate between different moods. There’s the version of me that wants to see a really moving documentary. And there’s the version that wants to zone out to a comedy. All of us have these moods, of course, and to predict what movie we would want to see would require knowing our mood at any given time. Your Netflix account only knows who you are, it doesn’t know which you is sitting down to watch a movie at any particular moment.
And all of this is not even considering the exponentially more difficult challenge of picking an optimal movie when you add a friend or spouse to the mix.
This is not to say that there’s no order or predictability to the movies you will like. When my wife recommends a movie to me, she is highly likely to be right. It’s just that she has a brain, which is able to perform calculations that are much more complex than any algorithm currently can.
