Is Science Stronger When It is Decentralized?

If you think of truth as acorns on the floor of a dark forest, then scientists can hunt for those acorns individually or in teams. Because academic science shares its knowledge, you'll often find scientists crowding around some trees where they think the acorns are more plentiful.

Not everything is an acorn though and scientists often fool themselves. The replication crisis is a symptom of that, but it's not really new. As Feynman put it, "Science is the belief in the ignorance of experts." In other words, truth doesn't come down from on high, it is a process fueled by doubt, exploring dark alleys, and testing each other's claims. Sometimes there are better acorns where no one is looking. Sometimes groups are gathered around a cat's litter box.

The question I'm interested in here is what is the best way to find high-quality acorns? Everyone can hunt independently or we can work together in close-knit teams. Independent, and therefore decentralized explorers, can find the same thing but may have different measures for discriminating real acorns from cat poo. On the other hand, if we hunt together, perhaps our cooperative wisdom may better avoid confirmation bias.

A recent article Danchev et al., (2018) explored this difference by investigating more than 50,000 published results of drug-gene interactions and examined their replication probability on various measures. They considered centralized efforts as multiple papers by authors who shared authorship in the past. Decentralized efforts were papers by authors who had not published together.

The key result for me is this: "Claims supported by many publications have about 45 percent higher probability to replicate when investigated by decentralized versus centralized communities (Figs. 3A and S8). Even if a claim garners wide support, if studied exclusively by a centralized scientific community, the claim is indistinguishable in replication probability from a claim reported in a single paper."

In other words, if we're to base claims on the weight of the evidence, we are better off considering studies by groups who have not worked together on this problem. A group of results all from the same community are not more reliable than one result by that community.

If you come across a group of people in the forest claiming they found acorns, they are less reliable if they all know each other well than if they don't.

Similar results have been shown for how groups of networked individuals solve problems in a variety of contexts (e.g., Barkoczi & Galesic, 2016). The general result is that when problems are hard because environments are complex, it is better to have inefficient communication.

The reason is that easy communication between individuals can lead to groups settling on suboptimal solutions. Inefficient communication creates a filter that information has to pass through before it reaches everyone else. This filter also gives others more time to explore before they start to feel the force of social proof, which can narrow the search too soon.

Follow me on twitter