Embroidered data: needle & thread not required

We’re all familiar with examples of research misconduct (Marc Hauser being a prominent recent example), but there are plenty of other less deliberate and more insidious ways science can lie to itself.  These include publication bias, choosing a method of statistical analysis that gives the desired answer, etc.  Those are worth discussing, but this post will focus on an informative and (to me anyways) humorous example of embroidered data, which is when a series of misrepresentations of a data set build upon themselves, with the end result and the inferences drawn from it having little connection to reality.  I feel such embroidery happens easily as we filter the literature through our biases and limited abilities of retention.  Most examples may not be as egregious as the following, but they are still failures of science to regulate itself.

THE KAIBAB DEER (this figure is taken from Colvinaux’s 1973 textbook, “Introduction to Ecology.”)
The Kaibab plateau is an area bordering the Grand Canyon that had undergone a series of disturbances from fires, sheep and cattle grazing and finally, predator removal (which occurred after it was designated a park by Teddy Roosevelt).

A subsequent increase in the deer population (Figure 1a) was documented by Rasmussen’s 1941 monograph, “Biotic Communities of the Kaibab Plateau, Arizona.”  The apparent increase was attributed to the removal of top predators.  The solid circles represent the park supervisors’ estimates, the open circles represent those of visitors to the park.  A contemporary wildlife biologist would probably roll his or her eyes at either method, but common sense would suggest that supervisors, who spend far more time in the park, would give more accurate estimates.  At the least, both estimates are represented and their sources noted in Rasmussen’s original paper.

"The Kaibab deer herd fiction; a history of embroidered data. (A) Population estimate of the Kaibab deer herd, copied from Rasmussen (1941). Linked solid circles are the forest supervisor's estimates' circles give estimates of other persons, and the dashed line is Rasmussen's own estimate of the trend. (B) A copy of Leopold's (1943) interpretation of the trend. © A copy of trend given by David Davis and Golley (1963), after Allee et al. (1949), after Leopold (1943) from Rasmussen (1941). (After Caughley, 1970.)

Aldo Leopold (yes, that Aldo Leopold) started the real trouble by basing a publication figure on the curve drawn to fit the visitors’ estimates (Figure 1b).  Two problems should be apparent, 1) the second, and presumably more accurate estimate is ignored and 2) he only reproduced the fitted curve drawn by Rasmussen, which is obviously not an actual best-fit curve as it is drawn to intersect the maximum.   Furthermore, the shape of the curve is altered:  the left-hand side of Leopold’s curve has a sigmoid shape suggestive of a population undergoing logistic growth.  This is what we would expect of a population released from a key restraint.  The right hand side shows a sharp decrease, characteristic of a population that has exceeded its environment’s carrying capacity.  These alterations suggest that the ecological ideas Leopold wished to illustrate biased his interpretation and reproduction of the data.

Finally, Leopold’s modifications were codified in Allee’s 1949 ecology textbook, “Principles of Animal Ecology (click here for the original figure) (Figure 1c).  His comments on the figure thus became accepted fact, while the data they were originally based on is completely obscured.

This data set is still considered a classic example of the control exerted by predators on prey abundance, as Wikipedia demonstrates.

So what can we take from this example of embroidery?  In a narrow sense, predators do not control prey abundance as closely as is commonly thought, as habitat recovery and mitigation of other anthropogenic disturbances probably had a larger effect in the case of the Kaibab deer. (here’s a badly scanned pdf of the chapter I took the figure from if you want more information).

The larger point is obvious: “Look at the data,” to quote my adviser who first showed me this figure.  Science works best when methodology is transparent and a cautious, sound interpretation of the data is suggested.  It also means that we must read the original papers that are the basis of the theory or phenomenon that we’re investigating.  Just about every new grad student has had that point made to them, followed by enough demands to make such historical literature (as arcane and opaque as they often are) the first thing triaged, , but it is necessary if science is to successfully regulate itself.

One response to “Embroidered data: needle & thread not required

  1. Not so fast. Colinvaux (and Caughley, and Botkin) did not do a very good job of detective work on the where/why behind Leopold’s selection of the data. By the time Leopold told the story, the Forest Service had revised its population estimates. Walter Mann was the ranger, and he published 3 versions of his report (Mann WG. 1931, 1941. The Kaibab deer: a brief history and present plan of management. USDA Forest Service, Kaibab
    National Forest, Fredonia, Arizona (original 1931 report
    amended 1934 and 1941). The 1941 version (two years before Leopold published his) stated: “Forest officers now believe that there were more than 100,000 deer in 1924” (that’s in contast to the earlier reports estimating 30,000 deer). Leopold was using the consensus values when he popularized his story. For more background, see Chris Young’s excellent book, Young CC. 2002. In the absence of predators: conservation and
    controversy on the Kaibab Plateau. Lincoln (NE): University
    of Nebraska Press.) Or a research article that examined whether the age structure of aspen (preferred deer browse) supported or refuted the story: Binkley, D., M. Moore, W. Romme, and P. Brown. 2006. Was Aldo Leopold right about the Kaibab deer herd? Ecosystems 9: 227–241 — downloadable: http://www.cnr.colostate.edu/%7Edan/papers/Ecosystems_2006_Kaibab.pdf

    The bottom line is that details are fundamentally important, both when weaving a new ecological story, and when critiquing old stories.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s