Macroscopic regularity over microscopic Brownian motion | the secret beneath the wisdom of crowds

Happy New Year! 2008 will be an exciting new year for many reasons. The philosophy of Web 2.0 has been understood and accepted by more and more people and organizations. "Engaging the wisdom of crowds" has been a slogan of many new-age startups, as well as a few old corporations. In the year 2008, we will watch more deep implementation of this philosophy in the industrial realm. On the other hand, the new concept "Giant Global Graph" starts to be a bridge connecting Web 2.0 and Semantic Web. As the result, Semantic Web, after many years research in labs, will gradually approaches the public audience in 2008. At last, in person I will start a new career in this year. 2008 thus means especially different to me.

In this first post at 2008, I want to address an interesting and essential topic about Web 2.0---why is the wisdom of crowds often superior to the wisdom of individuals even though the individuals might be domain experts and the crowd is generally unprofessional?

The previous argument is the foundation of a best-selling book The Wisdom of Crowds written by James Surowiecki. In his book, James describes many evidences to show the superiority of crowd wisdom and he has also suggested several ways to approach the crowd wisdom while at the same time avoiding some regular traps of misusing this concept. Nevertheless is the book well written, there are a few important absences in the book. One Amazon book reviewer Aaron Swartz criticized that James's book was lack of thoughtful analysis on the intrinsic reasons beneath the described phenomenon of crowd wisdom. A similar critique was made also by another Amazon book critic, David J. Gannon, who wrote that "[James's] choices seem to be crafted to provide maximum support while eliminating any element of contraindication whatsoever." Despite of my sincere support to the basic concept in James's book, I have to confess that I incline to the arguments made by the two critics. These negative viewpoints, however, does not decrease the value the book (they only mean that the book could be even better). But they really pointed out at least one essential missed issue in the book, i.e., the question I rose at the beginning: what are the intrinsic reasons beneath the fact that the wisdom of crowds is often superior to the wisdom of individuals? With this question, I read through the book carefully and finally, I got an insight---the superiority of the wisdom of crowds is another example of a natural fact that there is often macroscopic regularity over any microscopic irregularity.

One of the most famous irregular natural events is Brownian motion (click the picture on the left). Brownian motion is the random movement of particles suspended in a fluid or the mathematical model used to describe such random movements. In nature, Brownian motion exists everywhere. For example, in a body of water every individual H₂O molecule moves towards random directions and with varied velocities at the microscopic level regardless, however, how the entire body of water actually flows at the macroscopic level. An interesting phenomenon is that despite of the pure random movement of each individual H₂O molecule, a body of water always has its regular path of flow at the macroscopic level. Hence the irregularity of Brownian motion that may be supposed by many people to lead to random unpredictable flow actually generally cause regular predictable flow at the macroscopic level.

At the mathematical abstraction level, the phenomenon of Brownian motion and the phenomenon of crowd wisdom are indeed the same. In both cases, we have vectors that point to desultory directions and have random magnitudes in their respective directions. In Brownian motion these vectors represent the momentum of the particles and in crowd wisdom these vectors represent the decisions made by individual persons. By adding up these vectors, we may obtain a collective final result towards which the entire body moves. In Brownian motion the result is where the fluid body flows and in crowd wisdom it is what the collective decision is made by the crowd.

This mathematical model well explains several likely controversial claims in James's book. For example, in his book James observed that the collective decisions made by independent participants with highly diverse disciplinary areas is generally at least not worse than the collective decisions gathered by participants who are professional experts in the particular disciplinary area of the question. This observation is indeed surprising when we first see it because we often expect that to the same challenge the decision produced by a group of experts must be generally better than the decision made by a group of laymen. But based on the mathematical model of Brownian motion, we can see that although every individual particle acts purely random to each other, the sum of their momentum vectors always points to a fixed direction, i.e., the direction where the fluid body flows at the macroscopic level. In similar, although every individual person makes a decision solely upon his own profession that may be far away from the destinate disciplinary area, the sum of these decision vectors will point to a certain direction, i.e., the direction where the true answer sits (though nobody in the group really knows this direction). In this situation, it does not matter whether this group of people are domain experts or not. This is the myth and beauty of the wisdom of crowds.

The Brownian motion also explains why a group of laymen may even often outbeat a group of experts on producing a better collective decision. The figure on the left illustrate the idea. Above all, nobody truly knows the real direction of the goal. But experts often make decisions that are closer to the goal (this is why they are experts). By contrast, laymen often make decisions that are far away from the real goal. But if we sum up the decisions made by experts and the decisions made by laymen, the figure shows that it is not necessary that the collective decision of experts is better than the collective decision of laymen. Why? Experts are often biased in the same way, while laymen seldom have the type of biases experts have. Individual expert is certainly better than any individual layman on making professional decisions. But collectively experts often make biased decision---albeit the collective decision is also close to the real goal---since they are trained in similar ways. By contrast, the sum of layman's decisions might be more closer to the real goal since they do not have disciplinary bias in their mind. Though this vector addition diagram is simple, it shows why a group of laymen may beat a group of experts.

As James has emphasized in his book, independence is a crucial property of gathering better collective decisions, especially when these decisions are collected from a group of laymen. The new diagram at the left illustrates the reason. Unlike the previous one in which all experts and laymen make their decisions independently, in this new situation Expert 2 makes his decision after Expert 1 and so is Layman 2 after Layman 1. Humans are social creatures; so we often adjust our decisions to compromise the other persons in a group, even unconsciously. As the result, both Expert 2 and Layman 2 shift their decisions a little bit closer to the decision made by the respective former players. Immediately we see the consequence. The collective decision made by experts is still close to the goal since the decision made by Expert 1 is close to the goal. By contrast, the collective decision made by laymen starts to be away from the goal since the decision made by Layman 1 is away to the goal. This simple diagram shows why it is generally unconstructive (and often even destructive) to have a group of laymen communicate when they vote because most of the time they will unconsciously follow a direction that is far away from the real goal. By contrast, we may encourage the communication among experts since they are more likely to figure out a better solution after discussion. To the least, the worst expert decision may still be close to the goal.

Does the superiority of crowd wisdom suggest that we should not (or at least should not actively) hire experts on making decisions? The answer is no. There are two fundamental reasons why the existence of experts is actually crucial to the success of engaging the wisdom of crowds: (1) how to build a real diverse group of crowds and (2) how to aggregate the decisions made by the crowds.

As James has emphasized in his book, the property of diversity is critical to obtain high quality crowd-made decisions. In fact, this requirement of diversity is equivalent to the perspective of free of bias. When the voters in a group have diverse enough disciplinary backgrounds, disciplinary biases are eliminated to the least. Thus the aggregated collective decision would be closer to the real truth. But how to build a truly diverse set of participants is a problem. A random group of people invited from street might not necessary be a real diverse set to a particular question. To build a real diverse set requires highly professional experiences on the respective disciplinary field.

The two examples James told in the Introduction of his book are typical examples of why the construction of diverse groups is a highly professional job. In the first example, many people on the marketplace were beating on the weight of an ox. In his story, James addressed the participants as unprofessional normal people with respect to the issue of ox weight. Nevertheless was James right, these people were not so "unprofessional" as James had emphasized. We can safely assume that these people who had participated the beat regularly bought stuffs from the market. So they had basic knowledge on how to evaluate weight of varied things, even though they indeed were not professions on weighting oxes. This observation is important because it means that this group was really "diverse" with respect to the challenge. Think of repeating the same challenge among a group of first-grade elementary school students and we may see the difference between really "diverse" and fake "diverse" groups. By randomly calling up a group of first-grade elementary school students we may also have a diverse set, which is, however, not really "diverse" with respect to the demand of collecting crowd wisdom. The first-grade elementary school students are short of knowledge of weighting basic stuffs and thus the collective answer made by them is certainly "biased" by their short of knowledge.

Similar situation is for the second example, in which a group of professionals in varied fields was assembled by a naval official John Craven to guess the position of Scorpion, a US submarine disappeared in the North Atlantic. The story in the book was impressive; but would we be able to repeat this story by assembling a random group of professors at MIT? Certainly these professors must be brilliant and unquestionable experts in their disciplinary areas, but I bet they would certainly not able to guess the correct position of Scorpion, even collectively. Why? These professors generally have not been attended to the particular scenario and thus their decisions would be just little bit better than normal you and me in this case. On the contrary, the people John Craven had organized (as in the story) were the ones who were familiar to the submarine operation even though nobody had the complete knowledge of the particular case. So this group called by John Craven was a really diverse group and a random group of MIT professors is not.

Both the stories point out that assembling a "diverse" set is a highly professional request that demands the knowledge of domain experts.

Comparing to the demand of diversity, we may require more expertise on aggregating individual decisions made by a crowd to be a valuable collective decision. This type of aggregations is normally much more sophisticated than simply calculating the number of votes in different categories. As we show in the previous diagrams, the process of aggregating crowd wisdom is basically a procedure of vector addition (or more precisely a tensor addition since many times the number of dimensions would be more than three). How to divide a problem into varied dimensions and assign a measurement standard to it is a highly professional work that requires superior expertise on the disciplinary application area.

In summary, we now have a clear picture of how to take the benefit of crowd wisdom in real-world applications. First, we need a few experts. In contrast to rely on these experts to make decisions directly, however, we ask them to assemble a crowd that is truly diverse according to the problem. We let the crowd make their decisions independently. Then we ask the experts to aggregate the crowd decisions objectively based on the expertise of the experts. This is thus the procedure of engaging the wisdom of crowds.