don't make the same distinction, allowing luck to creep in if the researchers are not very careful in their methods. The difference in the quality of the findings is so dramatic that Ioannidis recommends a simple approach to observational studies: ignore them.19
The dual problems of bias and conducting too much testing are substantial, and by no means limited to medical research.20 Bias can arise from many factors. For example, a researcher who is funded by a drug company may have an incentive to find that the drug works and is safe. While scientists generally believe themselves to be objective, research in psychology shows that bias is most often subconscious and nearly unavoidable. So even if a scientist believes he is behaving ethically, bias can exert a strong influence.21 Furthermore, a bit of research that grabs headlines can be very good for advancing an academic's career.
Doing too much testing can cause just as much trouble. There are standard methods to deal with testing too much, but not all scientists use them. In much of academic research, scientists lean heavily on tests of statistical significance. These tests are supposed to indicate the probability of getting a result by chance (more formally, when the null hypothesis is true). There is a standard threshold that allows a researcher to claim that a result is significant. Here's where the trouble starts: if you test enough relationships, you will eventually find a few that pass the test but that are not really related as cause and effect.22
One example comes from a paper published in The Proceedings of the Royal Society B, a peer-reviewed journal. The article suggests that women who eat breakfast cereal are more likely to give birth to boys than girls.23 The paper naturally generated a great deal of attention, especially in the media. Stan Young, a statistician at the National Institute of Statistical Sciences, along with a pair of colleagues, reexamined the data and concluded that the finding was likely the product of chance as a result of testing too much. The basic idea is that if you examine enough relationships, some will pass the test of statistical significance by virtue of chance. In this case, there were 264 relationships (132 foods and two time periods), and the plot of expected values of statistical significance between the various relationships was completely consistent with randomness. Young and his collaborators conclude flatly that their analysis “shows that the [findings] claimed as significant by the authors are easily the result of chance.”24
So if we don't consider a sample that is large enough, we can miss the fact that a single strategy can always give rise to unanticipated results, as we saw in the case of the Sony MiniDisc. In contrast, we can comb through lots of possible causes and pick one that really has nothing to do with the effect we observe, such as women eating cereal and having boys as opposed to girls. What's common to the two approaches is an erroneous association between the effect, which is known, and the presumed cause. In each case, researchers fail to appreciate the role of luck.
Where Is the Skill? It's Easier to Trade for Punters Than Receivers
Many organizations, including businesses and sports teams, try to improve their performance by hiring a star from another organization. They often pay a high price to do so. The premise is that the star has skill that is readily transferable to the new organization. But the people who do this type of hiring rarely consider the degree to which the star's success was the result of either good luck or the structure and support of the organization where he or she worked before. Attributing success to an individual makes for good narrative, but it fails to take into account how much of the skill is unique to the star and is therefore portable.
Boris Groysberg, a professor of organizational behavior at Harvard Business School, has studied this topic in depth. His research shows that organizations tend to overestimate the degree to which the star's skills are transferrable. His most thorough study was of analysts at Wall Street firms.25 The primary responsibility of these analysts is to determine whether or not a given stock is attractive within the industry that they follow. (I used to be one of these analysts.) Institutional Investor magazine ranks the analysts annually, which provides a measure of quality.
Groysberg examined all of the moves by ranked analysts over a twenty-year period and found 366 instances of a star analyst moving to another firm. If the skill were associated solely with the analyst, you would expect the star's performance to remain stable when he or she changed jobs. That is not what the data showed. Groysberg writes, “Star analysts who switched employers paid a high price for jumping ship relative to comparable stars who stayed put: overall, their job performance plunged sharply and continued to suffer for at least five years after moving to a new firm.”26 He considered a number of explanations for the deterioration in performance and concluded that the main factor was that they left behind a good fit between their skills and the resources of their employer.
General Electric is a well-known source of managerial talent, and its alumni are disproportionately represented among CEOs in the S&P 500. Groysberg and his colleagues tracked the performance of twenty managers from GE that other organizations hired as chairman, CEO, or CEO-designate between 1989 and 2001. They found a stark dichotomy. Ten of the hiring companies resembled GE, so the skills of the executives were neatly transferable and the companies flourished. The other ten companies were in lines of business different from GE. For example, one GE executive went to a company selling groceries, whereas his experience had been in selling appliances. Even with a GE-trained executive at the helm, those companies delivered poor returns to shareholders. Again, developing skill is a genuine achievement. And skill, once developed, has a real influence on what we can do and how successful we are. But skill is only one factor that contributes to the end result of our efforts. The organization or environment in which a CEO works also has an influence. The evidence shows that employers systematically overestimate the power of an individual's skill and underestimate the influence of the organization in which he or she operates.
Along with some fellow researchers, Groysberg showed this point neatly by analyzing the performance of players who switched teams in the National Football League. They compared wide receivers with punters in the period between 1993 and 2002. Since each team has eleven players on the field at a time, wide receivers rely heavily on the strategy of the team and on interaction with their teammates, factors that can vary widely from team to team. Punters pretty much do the same thing no matter which team they play for, and have more limited interaction with teammates. The contrast in interaction allowed the scientists to separate an individual's skill from the influence of the organization on performance. They found that star wide receivers who switched teams suffered a decline in performance for the subsequent season compared with those who stayed with the team. Their performance then improved as they adjusted to their new team. Whether a punter changed teams or stayed put had no influence on his performance. Punters are more portable than wide receivers.27
As with testing too much or too little, the difficulty in determining the portability of a skill lies in the relationship between cause and effect. Groysberg's work dwells on stars and finds that the organizations that support them contribute meaningfully to their success. Yet we see people consistently overestimate skill in fields as diverse as catching touchdown passes and selling motorcycles.
Stories Can Obscure Skills
We re-create events in the world by creating a narrative that is based on our own beliefs and goals. As a consequence, we often struggle to understand cause and effect, and especially the relative contributions of skill and luck in shaping the events we observe.28 As we've seen, we may make the mistake of drawing conclusions from samples that are too small. We may fail to consider all of the causes that might lead to particular events. We might test too much—so much, in fact, that we wind up finding causes where we're simply seeing the results of chance. Or we may look at high performance and believe we are seeing a star with exceptional skill, when in reality we are seeing the combined effects of skill and the powerful influence that an organization can exert on someone. All of these mistakes are manageable, but it is critical to learn about them and to see where they apply if we are going to overcome them. The effort of untangling skill from luck, even with its practical difficulties, still yields great value when we are trying to improve the way we make decisions.
CHAPTER 3
THE LUCK-SKILL CONTINUUM
IN 2006, TRADINGMARKETS,