Identifying Wisdom in a Crowd When Problems Are Hard

Groups often contain wisdom. When the group's wisdom overshadows its collective ignorance, we call it the wisdom of the crowds.

But groups sometimes contain madness as well. Groups are often subject to polarization, social influence, and ingroup-outgroup thinking, which can make them unreliable, especially when problems are hard. How can we best access the wisdom of the crowd when there isn't much wisdom to access?

Suppose you want to know the capital of Pennsylvania and you have a group of 100 people to ask. If you ask them all, the most popular answer is likely to be Philadelphia. But if you use the most popular answer in this situation, you'll be wrong. The capital of Pennsylvania is Harrisburg.

So how can you get Harrisburg if most people think Philadelphia?

One often-attempted technique is to use confidence. Ask people how confident they are and then use this to weight the answers. More confident answers might be more correct.

In actual practice, however, confidence does not always correlate with knowledge. More ignorant people often have the highest level of confidence. This is called the Dunning-Kruger effect, and it should make us wary of the loudest voice in the room.

If confidence is not a good indicator of knowledge, what can we do? A recent article in Nature by Drazen Prelec provides a powerful answer.

First, ask people what they think. Then ask them what other people are most likely to think. Then take the difference.

Consider our Pennsylvania example. In this case, people who believe that Philadelphia is the capital are less likely to have good information about the plausible alternatives of others. So they will say Philadelphia when you ask them what others will say.

The people who say Harrisburg will know that the more common belief is Philadelphia. They probably even once thought that themselves.

By taking the difference between what people say and what they think other people will say, one arrives at a measure of Surprising Popularity. Philadelphia will be the answer that most people expect others to say, but it will be offered less frequently than expected because of all the Harrisburg believers. Harrisburg will be offered by a few as their own belief, but rarely offered as the expectation of others. It will therefore be surprisingly popular.

It's a clever solution to a problem that's been around for a long time.

Follow me on twitter