Why Do Algorithms Make Mistakes?

Two years ago, the real-estate company Zillow began a program to buy and sell homes. The idea of the program, Zillow Offers, was to use the company’s large amount of historical real-estate data to build extremely complex predictive models capable of quickly estimating the market value of homes. After filling out a short form, sellers received an offer within two business days.

Fast forward to last week. Zillow has announced that the program was experiencing huge losses, with their algorithm having been substantially overpaying for homes. As a result, the company is stopping the program and laying off a quarter of its workforce (2,000 out of 8,000 employees).

The story of Zillow Offers is a reminder that algorithms are as fallible as the humans who create them. As algorithms play an increasingly important role in business and everyday life, it’s becoming more important to understand when they are likely to fail.

There are three key reasons why predictive algorithms can make big mistakes.

1. The Wrong Data

An algorithm can only make accurate predictions if you train it using the right type of data. For example, the characteristics that make a house valuable in San Francisco (good public schools) might not matter as much in other parts of the country (in Dallas people care more about having large backyards). This means that if you build a model using only data on houses from San Francisco, it will do a bad job predicting how much houses are worth in Dallas (and vice versa).

The issue of training is particularly important when models are trying to predict outcomes involving people. In one famous example, Amazon built an algorithm to identify top talent in the resumes of job applicants. Amazon’s model was based on finding applicants who matched top employees currently working for the company. Because Amazon already had many successful men in top positions, the algorithm strongly preferred resumes from men and penalized resumes from women.

In order for an algorithm to succeed, creators need to provide it with the right training data.

2. The Wrong Outcome

Algorithms are often designed to predict one specific outcome. In marketing, this might mean predicting which version of an advertisement will get the most clicks from users. In human resources, algorithms might try to predict which employees will be productive and efficient in their jobs. The problem is that focusing exclusively on one outcome can lead to disastrous consequences. An algorithm designed to only identify the fastest and most efficient job applicants can create an organization full of antisocial and difficult-to-work-with employees. But most organizations want employees who are both efficient and easy to get along with.

Building a model to focus exclusively on one thing (maximizing product sales, or employee productivity) might lead to ignoring other important considerations.

3. Some Things Can’t Be Predicted

The last problem for algorithms is that some outcomes are just harder to predict than others. In particular, studies focusing on human relationships have found that algorithms struggle to predict romantic attraction and long-term relationship outcomes (Joel et al., 2017). An algorithm can learn how to reliably predict the weather, and master chess and Jeopardy! But there are some tasks, particularly social tasks, that remain elusive. So far, algorithms can't predict which TV shows will become successful, or who you will fall in love with.

Should You Trust Algorithms?

Despite the potential flaws in algorithmic decision making, research suggests that it is still usually better than human judgment (Dietvorst et al., 2015). Although algorithms aren’t perfect, it’s important to remember that the old way of making decisions (human intuition) can also be quite flawed.

The key might be to avoid letting human bias interfere with algorithmic decision making.