ection>
STAT-SPOTTING
STAT-SPOTTING
A FIELD GUIDE TO IDENTIFYING DUBIOUS DATA
UPDATED AND EXPANDED
JOEL BEST
University of California PressBerkeleyLos AngelesLondon
University of California Press, one of the most distinguished university presses in the United States, enriches lives around the world by advancing scholarship in the humanities, social sciences, and natural sciences. Its activities are supported by the UC Press Foundation and by philanthropic contributions from individuals and institutions. For more information, visit www.ucpress.edu.
University of California Press
Berkeley and Los Angeles, California
University of California Press, Ltd.
London, England
© 2008, 2013 by The Regents of the University of California
First paperback printing 2013
ISBN 978-0-520-27998-8
eISBN 9780520957077
“Crystal-Clear Risk” on p. 23 is from “Party, Play—and Pay: Multiple Partners, Unprotected Sex, and Crystal Meth,” Newsweek, February 28, 2005, 39. © 2005 Newsweek, Inc. All rights reserved. Used by permission and protected by the Copyright Laws of the United States. The printing, copying, redistribution, or retransmission of the Material without express written permission is prohibited.
Earlier discussions of some of the examples used in this book appeared in “Birds—Dead and Deadly: Why Numeracy Needs to Address Social Construction,” Numeracy 1, no. 1 (2008): article 6, and in the “Out of Context” feature in issues of Contexts magazine published between spring 2005 and fall 2007 (volumes 4–6).
The Library of Congress has catalogued an earlier edition as follows:
Library of Congress Cataloging-in-Publication Data
Best, Joel.
Stat-spotting : a field guide to identifying dubious data / Joel Best.
p. cm.
Includes bibliographical references and index.
ISBN 978-0-520-25746-7 (cloth : alk. paper)
1. Sociology—Statistical methods. 2. Social problems—Statistical methods. I. Title.
HM535.B4772008
301.072’7—dc222008017175
Designer: Nola Burger
Text: 11/15 Granjon
Display: Akzidenz Grotesk and Akzidenz Grotesk Condensed
Manufactured in the United States of America
21 20 19 18 17 16 15 14 13
10 9 8 7 6 5 4 3 2 1
The paper used in this publication meets the minimum requirements of ANSI/NISO Z39.48-1992 (R 2002) (Permanence of Paper).
The statistics you don’t compile never lie.
– STEPHEN COLBERT
CONTENTS
PREFACE TO THE 2013 EDITION
PART 1GETTING STARTED
A.SPOTTING QUESTIONABLE NUMBERS
B.BACKGROUND
PART 2VARIETIES OF DUBIOUS DATA
C.BLUNDERS
D.SOURCES: WHO COUNTED–AND WHY?
D.3Shocking Claims
D.4Naming the Problem
E.DEFINITIONS: WHAT DID THEY COUNT?
E.1Broad Definitions
E.2Expanding Definitions
E.3Changing Definitions
E.4The Uncounted
F.MEASUREMENTS: HOW DID THEY COUNT?
F.1Creating Measures
F.2Odd Units of Analysis
F.3Loaded Questions
F.4Raising the Bar
F.5Technical Measures
G.PACKAGING: WHAT ARE THEY TELLING US?
G.1Impressive Formats
G.2Misleading Samples
G.3Convenient Time Frames
G.4Peculiar Percentages
G.5Selective Comparisons
G.6Statistical Milestones
G.7Averages
G.8Epidemics
G.9Correlations
G.10Discoveries
H.RHETORIC: WHAT DO THEY WANT US TO THINK?
H.1Using Short-Term Turnover to Measure Long-Term Problems
H.2Sudden Turns for the Worse
H.3Designating Myths
H.4Rhetorical Flourishes
I.DEBATES: WHAT IF THEY DISAGREE?
I.1Causality Debates
I.2Equality Debates
I.3Policy Debates
PART 3STAT-SPOTTING ON YOUR OWN
J.SUMMARY: COMMON SIGNS OF DUBIOUS DATA
K.BETTER DATA: SOME CHARACTERISTICS
L.AFTERWORD: IF YOU HAD NO IDEA THINGS WERE THAT BAD, THEY PROBABLY AREN’T
M.SUGGESTIONS FOR THOSE WHO WANT TO CONTINUE STAT-SPOTTING
ACKNOWLEDGMENTS
NOTES
INDEX
PREFACE TO THE 2013 EDITION
I was introduced to thinking about bad statistics when I read Darrell Huff’s How to Lie with Statistics as a first-year college student. By the time I checked out the library’s copy, that book was more than ten years old, and the volume showed a lot of wear. Huff’s examples