We’re all familiar with examples of research misconduct (Marc Hauser being a prominent recent example), but there are plenty of other less deliberate and more insidious ways science can lie to itself. These include publication bias, choosing a method of statistical analysis that gives the desired answer, etc. Those are worth discussing, but this post will focus on an informative and (to me anyways) humorous example of embroidered data, which is when a series of misrepresentations of a data set build upon themselves, with the end result and the inferences drawn from it having little connection to reality. I feel such embroidery happens easily as we filter the literature through our biases and limited abilities of retention. Most examples may not be as egregious as the following, but they are still failures of science to regulate itself.
THE KAIBAB DEER (this figure is taken from Colvinaux’s 1973 textbook, “Introduction to Ecology.”)
The Kaibab plateau is an area bordering the Grand Canyon that had undergone a series of disturbances from fires, sheep and cattle grazing and finally, predator removal (which occurred after it was designated a park by Teddy Roosevelt).
A subsequent increase in the deer population (Figure 1a) was documented by Rasmussen’s 1941 monograph, “Biotic Communities of the Kaibab Plateau, Arizona.” The apparent increase was attributed to the removal of top predators. The solid circles represent the park supervisors’ estimates, the open circles represent those of visitors to the park. A contemporary wildlife biologist would probably roll his or her eyes at either method, but common sense would suggest that supervisors, who spend far more time in the park, would give more accurate estimates. At the least, both estimates are represented and their sources noted in Rasmussen’s original paper.
Aldo Leopold (yes, that Aldo Leopold) started the real trouble by basing a publication figure on the curve drawn to fit the visitors’ estimates (Figure 1b). Two problems should be apparent, 1) the second, and presumably more accurate estimate is ignored and 2) he only reproduced the fitted curve drawn by Rasmussen, which is obviously not an actual best-fit curve as it is drawn to intersect the maximum. Furthermore, the shape of the curve is altered: the left-hand side of Leopold’s curve has a sigmoid shape suggestive of a population undergoing logistic growth. This is what we would expect of a population released from a key restraint. The right hand side shows a sharp decrease, characteristic of a population that has exceeded its environment’s carrying capacity. These alterations suggest that the ecological ideas Leopold wished to illustrate biased his interpretation and reproduction of the data.
Finally, Leopold’s modifications were codified in Allee’s 1949 ecology textbook, “Principles of Animal Ecology (click here for the original figure) (Figure 1c). His comments on the figure thus became accepted fact, while the data they were originally based on is completely obscured.
So what can we take from this example of embroidery? In a narrow sense, predators do not control prey abundance as closely as is commonly thought, as habitat recovery and mitigation of other anthropogenic disturbances probably had a larger effect in the case of the Kaibab deer. (here’s a badly scanned pdf of the chapter I took the figure from if you want more information).
The larger point is obvious: “Look at the data,” to quote my adviser who first showed me this figure. Science works best when methodology is transparent and a cautious, sound interpretation of the data is suggested. It also means that we must read the original papers that are the basis of the theory or phenomenon that we’re investigating. Just about every new grad student has had that point made to them, followed by enough demands to make such historical literature (as arcane and opaque as they often are) the first thing triaged, , but it is necessary if science is to successfully regulate itself.