Bad Analysis

And Behind Door No. 1, a Fatal Flaw (New York Times)

2008-04-08T14:05:00.000-07:00

Really interesting article by John Tierney in the New York Times on cognitive dissonance.

"The Monty Hall Problem has struck again, and this time it’s not merely embarrassing mathematicians. If the calculations of a Yale economist are correct, there’s a sneaky logical fallacy in some of the most famous experiments in psychology."

Unintended Consequences (New York Times)

2008-01-27T07:08:00.000-08:00

Interesting article by Steven Levitt and Stephen Dubner about how well-meaning attempts to regulate complicated systems are likely to to have unintended consequences. Here is a short excerpt:

"Acemoglu and Angrist found that when the A.D.A. was enacted in 1992, it led to a sharp drop in the employment of disabled workers. How could this be? Employers, concerned that they wouldn’t be able to discipline or fire disabled workers who happened to be incompetent, apparently avoided hiring them in the first place.

How long have such do-good laws been backfiring? Consider the ancient Jewish laws concerning the sabbatical, or seventh year. As commanded in the Bible, all Jewish-owned lands in Israel were to lie fallow every seventh year, with the needy allowed to gather whatever food continued to grow. Even more significant, all loans were to be forgiven in the sabbatical. The appeal of such unilateral debt relief cannot be overestimated, since the penalties for defaulting on a loan at the time were severe: a creditor could go so far as to take a debtor or his children into bondage.

So for a poor Jewish sandal maker having trouble with his loan payments, the sabbatical law was truly a godsend. If you were a creditor, however, you saw things differently. Why should you lend the sandal maker money if he could just tear up the loan in Year Seven? Creditors duly gamed the system, making loans in the years right after a sabbatical, when they were confident they would be repaid, but then pulling tight the purse strings in Years Five and Six. The resulting credit drought was so damaging to poor people that it fell to the great sage Hillel to fix things."

Is a Carl Doomed To Be C Student?

2007-12-14T20:10:00.000-08:00

A study reported widely last month found that "students whose names start with the letters C or D, which denote mediocre marks in some grading systems, did not perform as well as other pupils with different initials."

As Carl Bialik points out in the Wall Street Journal, while the relationship that the researchers found was statistically significant, it doesn't mean it is important:

"University of California, Irvine, statistician Hal Stern points out something most media missed. The effect is tiny: 0.02 of a grade-point average point lower for the initials C and D (and this columnist isn't including that because of his first initial). Therein lies a lesson in the difference between statistical significance -- the confidence that there is some association between two factors -- and the strength of that association.

"In very large samples like the ones here, even small differences will be judged statistically significant," Prof. Stern says. "This means that we're confident the difference is not zero. It does not mean the difference we see is important." Prof. Nelson agrees that this effect is "so small that you shouldn't worry about it" when naming a child, though he does say the study exposes an example of how the unconscious mind can undermine conscious motivation.

But Bowling Green statistician Jim Albert warns: "You can prove any silly hypothesis ... by running a statistical test on tons of data.""

Citing Statistics, Giuliani Misses Time and Again

2007-11-30T18:28:00.000-08:00

New York Times article on Rudy Giuliani's record reporting the facts during his presidential campaign.

Four Pinocchios for Ron Paul (The Fact Checker)

2007-11-17T07:44:00.000-08:00

Good analysis of Ron Paul's claim that we could eliminate personal income taxes and still balance the budget.

Rudy Giuliani, amateur epidemiologist

2007-11-09T09:27:00.000-08:00

There have been many articles written about poor use of prostate cancer statistics by Rudy Giuliani. The main complaint is that he uses faulty logic to calculate prostate cancer survival rates. However, I think the bigger problem with Giuliani's numbers is that he cherry picks numbers to prove his points, providing no context. Once again, this appears to be a complicated issue and simplistic analysis by a politician.

Background
First, here's the background. According to a Giuliani radio ad:

"I had prostate cancer, five, six years ago. My chance of surviving prostate cancer, and thank God I was cured of it, in the United States, 82 percent. My chances of surviving prostate cancer in England, only 44 percent under socialized medicine."

As pointed out by factcheck.org and The Fact Checker at the Washington Post, the 44% figure was arrived at by simplistically dividing per capita prostate cancer mortality by per capita prostate cancer diagnoses (and subtracting that figure from 100%). Unfortunately, the people diagnosed in a given year are not the same people that die in that year, so you can't figure out what your odds of surviving prostate cancer by using this data. To determine survival rate, you need to follow the same population over a period of time (5-year survival rate is the standard metric). The 5-year survival rate in the U.K. is 74.4%.

The bigger problem
Clearly, it is troubling that a guy who wants to be part of the debate on the nation's health care doesn't have anyone on his staff that really understands the data. Equally troubling is that even after the doctors whose study he bases his claims pointed out his error, he continued to use the misleading numbers. But even if Giuliani used the proper numbers (which are 5-year survival rates of 74% in the U.K. and 98% in the U.S.), the conclusion he draws is simplistic at best.

Let me give you some statistics that would seem to refute Giuliani's conclusion about socialized medicine:

Average life expectancy in the U.K. is 79.4 years, compared to only 78.2 in the U.S.
Infant mortality rate in the U.K. is 5.01 per thousand, compared to 6.37 in the U.S.
Health care spending per capita in the U.K. is $1,675, compared to $4,271 in the U.S.

Does this mean that socialized medicine is better than private medicine? In my opinion, that would be an equally simplistic conclusion. We need to control for a lot of things in order for the data to be meaningful: population demographics, approach toward prevention, detection and treatment, etc. Unfortunately, intellectually honest analysis doesn't seem to be a Giuliani strong point.

Most Science Studies Appear to Be Tainted By Sloppy Analysis (Wall Street Journal)

2007-09-15T08:33:00.000-07:00

Interesting article by Robert Lee Holtz at the Wall Street Journal about mistakes in published research. He references the work of Dr. John Ioannidis, who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University. It also references an even more interesting paper he published in 2005.

From the article:

"Flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. 'There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims,' Dr. Ioannidis said. 'A new claim about a research finding is more likely to be false than true.'"

"The hotter the field of research the more likely its published findings should be viewed skeptically, he determined."

"Take the discovery that the risk of disease may vary between men and women, depending on their genes. Studies have prominently reported such sex differences for hypertension, schizophrenia and multiple sclerosis, as well as lung cancer and heart attacks. In research published last month in the Journal of the American Medical Association, Dr. Ioannidis and his colleagues analyzed 432 published research claims concerning gender and genes."

"'Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual,' Dr. Ioannidis said."

From the paper:
The paper lays out several factors that influence the probability of the results of a study being true:

Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true ... other factors being equal, research findings are more likely true in scientific fields that undertake large studies, such as randomized controlled trials in cardiology (several thousand subjects randomized) than in scientific fields with small studies, such as most research of molecular predictors (sample sizes 100-fold smaller).

Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true ... research findings are more likely true in scientific fields with large effects, such as the impact of smoking on cancer or cardiovascular disease (relative risks 3–20), than in scientific fields where postulated effects are small, such as genetic risk factors for multigenetic diseases (relative risks 1.1–1.5). Modern epidemiology is increasingly obliged to target smaller effect sizes. Consequently, the proportion of true research findings is expected to decrease.

Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true ... research findings are more likely true in confirmatory designs, such as large phase III randomized controlled trials, or meta-analyses thereof, than in hypothesis-generating experiments.

Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true ... Adherence to common standards is likely to increase the proportion of true findings. The same applies to outcomes. True findings may be more common when outcomes are unequivocal and universally agreed (e.g., death) rather than when multifarious outcomes are devised (e.g., scales for schizophrenia outcomes). Similarly, fields that use commonly agreed, stereotyped analytical methods (e.g., Kaplan-Meier plots and the log-rank test) may yield a larger proportion of true findings than fields where analytical methods are still under experimentation (e.g., artificial intelligence methods) and only “best” results are reported.

Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true ... Prejudice may not necessarily have financial roots. Scientists in a given field may be prejudiced purely because of their belief in a scientific theory or commitment to their own findings. Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifications for promotion or tenure.

Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true ... With many teams working on the same field and with massive experimental data being produced, timing is of the essence in beating competition. Thus, each team may prioritize on pursuing and disseminating its most impressive “positive” results. “Negative” results may become attractive for dissemination only if some other team has found a “positive” association on the same question. In that case, it may be attractive to refute a claim made in some prestigious journal. The term Proteus phenomenon has been coined to describe this phenomenon of rapidly alternating extreme research claims and extremely opposite refutations. Empirical evidence suggests that this sequence of extreme opposites is very common in molecular genetics.

The Media and the Median (Fallacy Files)

2007-08-26T16:01:00.000-07:00

Fallacy Files points out an amusing error in a recent New York Times science article, "The Myth, The Math, The Sex" by Gina Kolata. The article states:

"One survey, recently reported by the federal government, concluded that men had a median of seven female sex partners. Women had a median of four male sex partners. Another study, by British researchers, stated that men had 12.7 heterosexual partners in their lifetimes and women had 6.5.

But there is just one problem, mathematicians say. It is logically impossible for heterosexual men to have more partners on average than heterosexual women. Those survey results cannot be correct."

However, as Fallacy Files points out, even in a population of equal numbers of men and women over a given time period, the median number of sex partners could be different. All you need are a few very active women who raise the mean a lot but don't raise the median too much (Madonna-whore dichotomy?). From Fallacy files:

"It's discouraging that an article this innumerate would be published in the Times, and I suppose that the mathematician quoted was not the source of the confusion, but that it must have been introduced in the writing or editing. As it is, there's no evidence in the article of anything impossible in the statistics cited. A British survey is quoted, but the article doesn't indicate whether the numbers are medians, means, or what―which is a problem in itself!

Moreover, the article presents itself as busting the "myth" that heterosexual men are more promiscuous than heterosexual women. Surely, the "myth" is that the typical man has more sex partners than the typical woman. In order for this to be true, there must be some atypically promiscuous women. Whether the "myth" is true or not, I don't know―damn it, Jim, I'm a logician, not a sociologist!―but I do know that it is an empirical question and not a logical or mathematical one."

Issues are complicated. Politicians are not.

2007-06-24T13:45:00.000-07:00

Some good recent posts from factcheck.org on the 2008 presidential campaign. Here are a few excerpts:

How Giuliani made falling adoptions seem to rise using cherry-picked statistics: "In an earlier article we criticized Rudy Giuliani for saying adoptions went up 65 to 70 percent when he was mayor, when in fact adoptions at the end of his tenure were only 17 percent higher than at the start, and falling. His campaign still insists his claim is justified and offers its own interpretation of the statistical record. In this article we offer the former mayor's rationale, along with why we believe it is a classic case of how candidates and public officials sometimes use data selectively to create a false impression."

The leading three Democratic presidential candidates wax optimistic about ethanol. We provide a reality check: "Gas prices have hit record highs this year as 2008 presidential candidates outline their hopes for renewable fuels. In this story, we take a look at the reality. We focus specifically on E85, a popular ethanol-gasoline fuel blend, and the top three Democratic candidates' statements about this fuel as they fight to win votes in Iowa. We find that there are many technological bridges left to be crossed before E85, or other renewable fuels, can fulfill the role these candidates envision for them — or can start saving individual consumers cash at the pump."

Sen. John McCain has said President Bush's tax cuts have increased federal revenues. But revenues would have been even higher without them: "Republican presidential candidate Sen. John McCain has said that the major tax cuts passed in 2001 and 2003 have "increased revenues." He also said that tax cuts in general increase revenues. That’s highly misleading."

Dems Debating, the Sequel: "Former Sen. John Edwards of North Carolina talked about gas price manipulation by Big Oil where investigators have found none."

Third Time 'Round for GOP Hopefuls: "Former Massachusetts Gov. Mitt Romney committed the biggest factual fouls of the night, misleadingly asserting: That we went to war in Iraq because Saddam Hussein refused to allow weapons inspectors to come in; That there's an ocean of difference between his Massachusetts health plan and those "government takeover" plans of "every Democrat" running for president and; That Russia's income from oil exports is vastly larger than it actually is."

The Difficulty of Counting Divorces (The Numbers Guy)

2007-05-22T21:25:00.000-07:00

From The Numbers Guy (Carl Bialik) at the WSJ, an interesting look at some misleading reporting on divorce rates. The headline of an AP article says that "U.S. divorce rate lowest since 1970" and goes on to say that "despite the common notion that America remains plagued by a divorce epidemic, the national per capita divorce rate has declined steadily since its peak in 1981 and is now at its lowest level since 1970."

However, as Bialik points out, while the number of divorces per capita has gone down, so have the number of marriages per capita. So it isn't clear that the "divorce rate" as most people would define it (the chances any given marriage will end in divorce) has gone down. The misuse of this statistic by the AP article is doubly infuriating becase 1) it leads the uncritical reader to reach a potentially wrong conclusion, and 2) the question of whether divorce is more or less of a problem seems answerable.

The problem with getting a clear answer to that question is that the US Government no longer tracks detailed data on individual marriages and their outcomes. However, the crude per capita data that the AP misused can still be used to give us a hint at the answer. If you divide the divorces per capita in 2006 by marriages per capita, you get 49%, which is the source of the "50% of marriages end in divorce" stat. This number hasn't changed much since it shot up in the 1970s (in 1976 it was 51% and in 1966 it was 26%). However, the people getting divorced in a given year aren't the same ones getting married, so this number is flawed. (Although my wife pointed out to me that Renee Zellweger recently got married and divorced within the same year).

So instead, lets lag the divorce data by, say, 5 years. This seems pretty reasonable since the average marriage ending in divorce lasts ~7 years. Dividing 2006 divorces per capita by 2001 marriages per capita, we get a 46% divorce rate (closer to the number experts think is more likely correct). Again, going back to 1976, the divorce rate using this calculation was 47%.

While this analysis is very simplistic, it hardly paints a picture of steadily declining divorce rates. Here are some headlines for the AP article I think would be more appropriate:

"U.S. Divorce rate pretty much the same as it was 30 years ago"

"Americans marrying less, but not doing a much better job of choosing spouses they will stay married to"

"U.S. Marriage rate at its lowest point in the history of the country"

"U.S Divorce rate still highest in the world"

Drinking age paradox?

2007-05-03T08:17:00.000-07:00

I don't always agree with George Will, but his recent article on whether 21 is the appropriate drinking age started with me nodding my head in agreement:

"Public policy often illustrates the law of unintended consequences. Society's complexity -- multiple variables with myriad connections -- often causes the consequences of a policy to be contrary to, and larger than, the intended ones."

However, the rest of the article, whose premise is that that the legal drinking age should be lowered, is filled with half-baked arguments and numbers without context. The basic argument is that lowering the legal drinking age to 18 will help 18-21 year olds become more responsible drinkers and reduce the incidence of binge drinking and alcohol-related deaths. Will's op-ed piece as well as articles at Reason and National Review put forward some of the intellectually dishonest arguments in favor of lowering the drinking age:

Reason: "Oddly enough, high school students in much of the rest of the developed world — where lower drinking ages and laxer enforcement reign — do considerably better than U.S. students on standardized tests."

Is he seriously arguing that a lower drinking age causes higher standardized test scores? I think there may be some other variables at play here. The author either 1) is a moron or 2) thinks the people reading his article are morons.

National Review: "When it comes to alcohol, the United States is more like Indonesia, Mongolia, and Palau than the rest of the world: It is one of just four countries that requires people to be at least 21 years old to buy booze. The only countries with stiffer laws are Islamic ones."

Good argument. If Islamic countries do something, it must be wrong.

George Will: "The drinking age of 21 was one of 39 measures proposed during the 1980s by a presidential commission on drunken driving; various measures adopted did dramatically reduce the problem. But according to the National Institute on Alcohol Abuse and Alcoholism, about 5,000 people under 21 die every year from vehicular accidents, other injuries, homicides and suicides involving underage drinking."

So if the measures reduced the problem, why do you want to roll one of them back? Is 5,000 less than it used to be? If so then you should be looking for ways in addition to the current measures to improve on this number.

Haven't there been academic studies on the link between the legal drinking age and drunk-driving deaths?

Yes! Hundreds in fact. But apparently these authors didn't do the 5 minutes of research it would have taken to uncover some of them. This survey of 241 studies, from the University of Minnesota showed that "the preponderance of evidence indicates there is an inverse relationship between the MLDA and two outcome measures: alcohol consumption and traffic crashes." In other words, higher minimum legal drinking ages are correlated with less alcohol consumption and fewer traffic crashes.

The Secondary Effects of War

2007-04-14T14:41:00.000-07:00

One of the causes of poor analysis is ignoring or discounting the secondary effects of an action. Once classic example is price wars, which are usually started when one company tries to gain market share and doesn't accurately predict what its competitors will do in response.

I think one of the reasons the Iraq war has been so unsuccessful is that the secondary effects of our actions were underestimated or not seriously considered. The recent Red Cross report does a good job pointing out not only the "Shootings, bombings, abductions, murders, military operations and other forms of violence [which] are forcing thousands of people to flee their homes and seek safety elsewhere", but some of these secondary effects of war:

Health Care: "Health-care facilities are stretched to the limit as they struggle to cope with mass casualties day-in, day-out. Many sick and injured people do not go to hospital because it’s too dangerous, and the patients and medical staff in those facilities are frequently threatened or targeted."

Food: "Food shortages have been reported in several areas. According to the Iraqi Red Crescent, malnutrition has increased over the past year. The vastly inadequate water, sewage and electricity infrastructure is presenting a risk to public health."

Poverty: "Unemployment and poverty levels are rising and many families continue to rely on government food distributions to cover their immediate needs. According to government sources, an estimated one third of the population lives in poverty, while over five percent live in extreme poverty."

Physical Infrastructure: "Much of Iraq’s vital infrastructure is in a poor state of repair owing to lack of maintenance and because security constraints have impeded repair work on electrical power grids, water and sanitation systems, medical facilities and other essential facilities. Power shortages are growing worse throughout the country, including northern areas, owing largely to the failure to carry out maintenance and to increase generation capacity. Fuel shortages affecting power stations and acts of sabotage are further aggravating the crisis. As a result, water treatment plants, primary health-care centres and hospitals rely mainly on back-up generators, which often break down owing to excess usage or fall victim to the chronic fuel shortages."

Of course we aren't bombing hospitals and power plants, but our actions have indirectly caused these problems. If the proper homework was done, many of these secondary effects could have been accounted for, and we could have weighed them into our decision making process. The controversial Lancet study, which accounted for some of the human costs of the problems above, estimated 600,000 Iraqi's are dead because of our actions. I think that if we had explicitly considered these costs in our pre-war analysis, the decision we made would have been very different.

Greenpeace Bites Apple (STATS.org)

2007-04-06T16:38:00.000-07:00

Greenpeace recently published a report criticizing technology companies for their "green" practices and singling out Apple in particular. However, their criticism doesn't appear to have much data behind it, as pointed out by STATS.org. Here are a few excerpts:

"The evidence appears to be that as Apple doesn’t reclaim its unwanted or out-of-date computers, and because computers are made with chemicals that can be toxic, it’s a disgrace."

"The problem is that Greenpeace doesn’t seem to want to prove its case with actual epidemiological or toxicological data. It simple asserts that

“Right now, poison Apples full of chemicals (like toxic flame retardants, and polyvinyl chloride) are being sold worldwide. When they're tossed, they usually end up at the fingertips of children in China, India and other developing-world countries. They dismantle them for parts, and are exposed to a dangerous toxic cocktail that threatens their health and the environment.”

"It requires a little more digging to find Apple’s actual scorecard, wherein one finds that Greenpeace don’t actually measure the amount of “toxic” chemicals in Apple’s products."

"It’s easy to dismiss Greenpeace’s campaign as a cheap, self-serving publicity stunt, but it’s a stunt that has succeeded, if you look at the massive and largely uncritical media coverage. And none of the news reports critically examined the scientific basis for Greenpeace’s claims, the methodology of its ranking – or the ethics of accusing a company of being an environmental polluter without providing actual proof that there is real exposure and real harm. Apple deserves better – and so do we."

The Decoy Effect (Department of Human Behavior)

2007-04-06T16:06:00.000-07:00

Another interesting article from Shankar Vedantam, this time about biases in decision making when faced with more than two choices. Here are a few excerpts:

"Front-runners are usually focused on racing each other. They often do not realize that when people cannot decide between two leading candidates -- and it doesn't matter whether we are talking about politicians or consumer appliances -- our decision can be subtly swayed by whoever is in third place."

"Joel Huber, a Duke University marketing professor, showed how the decoy effect works with restaurants. Huber asked people whether they would prefer to eat at a five-star restaurant that was far away or at a three-star restaurant nearby. As with many choices in life, each restaurant had different advantages. If the better restaurant was also nearby, there would be no dilemma. But the question forced people to compare apples and oranges -- trade off quality against convenience -- which ensured no automatic answer."

"The human brain, however, always seeks simple answers. Enter the third candidate. Huber told some people there was also a choice of a four-star restaurant that was farther away than the five-star option. People now gravitated toward the five-star choice, since it was better and closer than the third candidate. (The three-star restaurant was closer, but not as good as the new candidate.)"

"Many people lavished hate on Ralph Nader for presumably taking votes away from the Democratic front-runner in the 2000 presidential election," said Scott Highhouse, who has studied the decoy effect at Bowling Green State University. "Research on the decoy effect suggests that Nader's presence, rather than taking votes away, probably increased the share of votes for the candidate he most resembled."

Department of Human Behavior

2007-03-25T08:59:00.000-07:00

I recently discovered this Washington Post column (thanks to Fallacy Files) called Department of Human Behavior by Shankar Vedantam. The column touches on many of the human psychological biases that can lead to errors in decision making. Here are a few examples of articles he has written about the Iraq war:

Disagree About Iraq? You're Not Just Wrong -- You're Evil

"Opponents of the war believe passionately that President Bush, his neoconservative allies and a complicit Congress deliberately misled the nation into war. Supporters of the president and the war concede that mistakes were made, especially on the question of whether Saddam Hussein had weapons of mass destruction, but say this involved no attempt to hoodwink the nation."

"What is interesting about the clash from a psychological perspective is not that supporters and critics disagree, but that large numbers of people on both sides claim to know the motives of people who disagree with them. When was the last time you heard people say that those who disagree with them on the Iraq war are well-meaning, smart, informed and thoughtful?"

"Nearly three-quarters of the people who supported the war believed that Bush was thinking about self-defense when he launched the invasion of Iraq. By contrast, fewer than 2 in 5 Americans who opposed the war were willing to grant that Bush was thinking of self-defense. Fully 70 percent of the people who supported the war said Bush was aiming to do good; only 27 percent of people who opposed the war believed that the president's motives were about doing good."

Iraq and the Danger of Psychological Entrapment

"When you invest yourself in something, it is exceedingly difficult to discard your investment. What is devilish about entrapment is not just that it can result in ever greater losses, but that those losses get you ever more entrapped, because now you have even more invested."

"[Social psychologist Scott] Plous said his alarm bells went off when he realized that President Bush was explicitly using the language of entrapment in speeches to rally support for the war. "Retreating from Iraq would dishonor the service of our brave men and women who have sacrificed in that country and have given their lives in that country, which would mean their sacrifice would be in vain," the president said recently."

""Rational decision-making should not be driven primarily by recovery of past costs," Plous said. "If you can no longer justify it in terms of what it will bring in the future and what its realistic prospects are, that is a warning sign you may have become entrapped.""

Iraq War Naysayers May Have Hindsight Bias

"Liberals' assertion that they 'knew all along' that the war in Iraq would go badly are guilty of the hindsight bias," agreed Hal Arkes, a psychologist at Ohio State University, who has studied the hindsight bias and how to overcome it. "This is not to say that they didn't always think that the war was a bad idea."

"Baruch Fischhoff, a psychologist at Carnegie Mellon University and a pioneer in the field of hindsight bias, found that Americans who made estimates about their danger after the Sept. 11, 2001, attacks recalled having made much lower estimates of risk a year later, after their fears failed to materialize."

"Indeed, research by both Fischhoff and Arkes show that people can fight the hindsight bias only when they honestly and systematically try to explain how different outcomes are possible. Such self-doubt is the exact opposite of how modern politics works: In the age of the blogosphere, certitude is king."

Blurb Watch (Fallacy Files)

2007-03-10T17:39:00.000-08:00

If you haven't seen it before, fallacyfiles.org does an occasional feature on "blurbs" from movie review taken out of context. Here are a few good ones (1, 2, 3) , and here is a sample of the most recent:

"An ad for [Norbit] has the following blurb:

"Eddie Murphy's comic skills are immense."
―Michael Wilmington, CHICAGO TRIBUNE

If you suspect that some context is missing, you're right:

Murphy's comic skills are immense, and "Dreamgirls" shows he's a fine straight dramatic actor too. So why does he want to make these huge, belching spectaculars, movies…swollen, monstrous and full of hot air…? "Norbit" makes you long for the days of "Beverly Hills Cop," when Murphy was lighter on his feet, and his movies were too."

Public school teachers underpaid?

2007-02-04T20:00:00.000-08:00

Interesting op-ed peice from the Wall Street Journal about public school teacher salaries based on this report. The basic idea is that, when looked at on an hourly basis, public school teacher salaries are good compared to other professions, which goes against the conventional logic. A few excerpts:

"According to the Bureau of Labor Statistics, public school teachers earned $34.06 per hour in 2005, 36% more than the hourly wage of the average white-collar worker and 11% more than the average professional specialty or technical worker."

"In the popular imagination, however, public school teachers are underpaid. "Salaries are too low. We all know that," noted First Lady Laura Bush, expressing the consensus view. "We need to figure out a way to pay teachers more.""

"Of course, public school teacher earnings look less impressive when viewed on an annual basis than on an hourly basis. This is because teachers tend to work fewer hours per year, with breaks during the summer, winter and spring. But comparing earnings on an annual basis would be inappropriate when teachers work significantly fewer hours than do other workers. Teachers can use that time to be with family, to engage in activities that they enjoy, or to earn additional money from other employment. That time off is worth money and cannot simply be ignored when comparing earnings. The appropriate way to compare earnings in this circumstance is to focus on hourly rates."

"Moreover, the earnings data reported here, which are taken directly from the National Compensation Survey conducted by the Bureau of Labor Statistics, do not include retirement and health benefits, which tend to be quite generous for public school teachers relative to other workers. Nor do they include the nonmonetary benefit of greater job security due to the tenure that most public school teachers enjoy."

Making bad choices (Social Science Statistics Blog)

2007-02-03T15:08:00.000-08:00

Interesting post from the Social Science Statistics Blog. It talks about a study which shows that the order in which you present information about various options can affect the decisions people make, including chosing the inferior option. A few excerpts:

"Researchers presented subjects with a set of restaurants and attributes to determined their objective "favorite." Then, two weeks later, they brought the same subjects in again and presented them with the same restaurants. This time, though, they had determined -- individually, for each subject -- the proper order of attributes that would most favor choosing the inferior alternative."

"Basically what they did is picked the attribute that most favored the inferior choice and put it first, hoping to establish that the inferior choice would get installed as the leader. The attribute that second-most favored the inferior choice was last, to take advantage of recency effects. The other attributes were presented in pairs, specifically chosen so that the ones that most favored the superior alternative were paired with neutral or less-favorable ones (thus hopefully "drowning them out.")"

"The results were that when presented with the information in this order, 61% of people chose the inferior alternative. The good news, I guess, is that it wasn't more than 61% -- some people were not fooled -- but it was robustly different than chance, and definitely more than you'd expect (since, after all, it was the inferior alternative, and one would hope you'd choose that less often). Moreover, people didn't realize they were doing this at all: they were more confident in their choice when they had picked the inferior alternative. Even when told about this effect and asked if they thought they themselves had done it, they tended not to think so (and the participants who did it most were no more likely to think they had done it than the ones who didn't)."

Going to college may be bad for your brain?

2007-01-20T10:33:00.000-08:00

From Statistical Modeling, Causal Inference, and Social Science, a post about an article from MSNBC. The article, titled "Higher education tied to memory problems later, surprising study finds", states that:

"Going to college is a no-brainer for those who can afford it, but higher education actually tends to speed up mental decline when it comes to fumbling for words later in life."

However, from the post:
"Looking at the article, we see "an older adult with 16 years of schooling or a college education scored about 0.4 to 0.8 points higher at baseline than a respondent with only 12 years of education." Based on Figures 1 and 2 of the paper, it looks like higher-educated people know more words at all ages, hence the title of the news article seems misleading."

How to read a press release (Fallacy Files)

2007-01-20T10:26:00.000-08:00

Interesting post from Fallacy Files. It shows how two completely accurate explanation of a study's finding can have a ver different impact on how it is interpreted by readers. From the post:

"One of the conventions of language use is that the order in which facts are related parallels the order in which they occurred. Another convention is that facts that are mentioned together should be relevant to one another, and a causal relationship is one form of relevance. For instance, "Mary got pregnant and got married" tells a very different story than "Mary got married and got pregnant". In the first case, you might well assume that the pregnancy causally contributed to Mary's decision to wed."

Taking Another Look At Murder Statistics (The Numbers Guy)

2007-01-20T10:12:00.000-08:00

Interesting article from The Numbers Guy about murder statistics. He points out two problems with relying on this statistic to judge the safety or police force effectiveness of a city:

1) The difference between murder and assualt is often the effectiveness of medical treatment. From the article:

"Prof. Harris was the lead author of a study in 2002 examining the decline in the murder rate between 1960 and 1999. Prof. Harris focused on the role improved medical care had on "lethality," which is the proportion of violent crimes that result in death. Even though the number of potentially deadly attacks surged in that time period, and the proportion of attacks involving guns also rose, lethality actually decreased by 70% -- to under 2% from nearly 6%. He concluded that improved medical-response times and trauma surgery were responsible for turning many would-be murders into assaults.

To demonstrate the effect of medical care, the researchers compared counties nationwide by various medical criteria, and found, for instance, that counties with at least one hospital were associated with 24% lower lethality than those without a hospital. Prof. Harris also found other medical factors that contributed to a drop in lethality of violent crimes, including the addition of physicians to a county's population and the presence of a facility for performing open-heart surgery."

2) The circumstances of a city have a big impact on violent crime. From the article:

"Think of a comparison between two hospitals' death rates (a topic I wrote about last year). If Hospital A has a higher proportion of patients suffering from life-threatening diseases, it wouldn't be fair to compare its death rate to Hospital B, which tends to treat patients with milder complaints. Prof. Friedmann's team believes that certain cities (Atlanta, Detroit) have higher underlying risk of murder than others (Denver, San Francisco).

The group relies on stats from the federal government, so its most recent ranking is for 2004. According to that ranking, San Francisco was No. 1 among 67 cities in adjusted homicide rates, despite ranking 30th in the raw rates. Atlanta, conversely, fell to No. 46 from No. 7 after the adjustments. And Detroit dropped to No. 37 on the list from No. 3 -- suggesting that the city's demographic profile helps explain what the Detroit News earlier this month called an "abysmal" crime picture for the city."

Linguistic Nonsense

2007-01-06T15:34:00.000-08:00

Fresh Air on NPR did an interesting piece about the book The Female Brain by Louann Brizendine. You can listen to the piece here or read the transcript. A few excerpts:

"Psychiatrist Louann Brizendine ... argues that most of the cognitive and social differences between the sexes are due to differences in brain structure. It's a controversial thesis. The New York Times's David Brooks and others have hailed the book as a challenge to feminist dogma, and Brizendine herself has charged that her critics are angry because her conclusions aren't politically correct. Actually, though, you can leave out the "politically" part."

"For example, Brizendine asserts that differences between men's and women's brains make women more talkative than men, and goes on to say that women on average use 20,000 words a day while men use only 7000. That factoid conforms so neatly with gender stereotypes about chatty women and taciturn men that a lot of people were indignant that anybody would spend money to discover anything so obvious. One reporter at a San Francisco TV station began his story on Brizendine by saying "Here's a news flash. Women talk more than men. Duh." Except that, duh!, it isn't true. It turns out that the figures Brizendine reported had been taken from a book by a self-help guru who had simply pulled them out of the air. And the studies that have been done generally show either that men talk slightly more than women or that the two sexes talk about the same amount."

"Or take Brizendine's claim that women on average speak twice as fast as men do. That's another cherished bit of gender lore, but no research shows anything of the sort -- the best evidence indicates that men on average speak a bit faster than women do. Nor is there any scientific basis for her claims that men think about sex every 53 seconds while women think about sex only once a day, or that women are more emotionally attentive because their more sensitive hearing enables them to hear subtle tones and nuances in speech that escape men."

"In short, saying that Brizendine's claims about sex differences in language are not exactly scientific gives "not exactly" a bad name."

Prioritizing the world's biggest problems

2007-01-06T15:25:00.000-08:00

Interesting lecture by Bjorn Lomborg, author of The Skeptical Environmentalist, about the need to prioritize the world's most important problems. The latest Copenhagen Consensus rankings are listed here.

Economist Bjorn Lomborg makes a persuasive case for prioritizing the world's biggest problems, asking "If we had $50 billion to spend over the next four years to do good in the world, where should we spend it?" His recommendations - based on the findings of the 2004 Copenhagen Consensus - controversially place global warming at the bottom of the list (and AIDS prevention at the top). Lomborg was named one of the 100 Most Influential People by Time magazine after the publication of his controversial book, The Skeptical Environmentalist which challenged widely-held beliefs that the environment is getting worse. Now the Danish economist is taking on the world's biggest problems with his Copenhagen Consensus. (Recorded February 2005 in Monterey, CA. Duration: 17:27)

The 2006 Dubious Data Awards (STATS.org)

2007-01-06T15:00:00.000-08:00

From STATS.org, the worst science stories of 2006:

Common Mistakes in Interpreting Statistics

2007-01-01T09:16:00.000-08:00

Very interesting (though long) lecture by statistician Peter Donnelly with several examples about mistakes most people make in interpreting statistics.

Statistician Peter Donnelly explores the common mistakes humans make in interpreting statistics, and the devastating impact these errors can have on the outcome of criminal trials. Donnelly is a statistics professor at Oxford University who collaborates with biologists, applying statistical models to genetics, with the hope of shedding more light on evolutionary history and the structure of the human genome (Recorded July 2005 in Oxford, UK. Duration: 22:06)

Bird flu death rate

2007-01-01T08:37:00.000-08:00

Interesting article from the Weekly Standard about the reported fatality rate from the H5N1 (bird flu) virus. As an example, this article from the Washington Post says that the "fatality rate for the H5N1 virus is about 60 percent."

The problem with this estimate is that it is based only on people who went to the hospital, presumably the most severe cases. So if an equal number of people didn't go to the hospital, the reported death rate would be 100% too high. Research has actually been done which can give us a better estimate. From the article:

"Over a three-month period in 2004, Swedish and Vietnamese researchers studied 45,478 residents in a rural district in Vietnam that had H5N1 outbreaks to find out how many had contact with sick birds and how many had flu-like illnesses. They published their results in January 2006 in the Archives of Internal Medicine. They found that of 8,149 who had a flu-like illness, 650 to 750 probably caught it from birds. Yet for all of 2004, the World Health Organization data indicated only 29 Vietnamese cases with 20 deaths. Thus what might seem to be a horrific mortality rate of almost two in three, or 69 percent, appears to be actually around one in 140 or 0.71 percent. This, in the rural portion of a Communist country with a state-run medical system. That 0.71 percent is in the same range as seasonal human flu."

The 5 Worst Predictions from 2006

2006-12-31T08:29:00.000-08:00

The end of the year is high-season for predictions about the next year. Unfortunately, the so-called experts rarely get called out for their bad predictions (often based on bad or no analysis). Here are 5 predictions from last year that should give you pause about the predictions you see for 2007.

5. Elections: Throughout the course of the midterm election campaigns, polls and predictions were everywhere. ThinkProgress.org highlights statements from Karl Rove, Michael Novak, Dick Cheney, George Will, and others who predicted a good day for Republicans.

4. Weather: In May, the National Oceanic and Atmospheric Administration forecast an 80% chance of an "above-normal hurricane season" this year. It turns out that it was a below-normal season. To be sure, all forecasts have a confidence interval, and at least they gave a 5% chance in their forecast of a below-normal season. However, I'm including this in the list because of the over-confident statement from the same report saying "the main uncertainty in this outlook is not whether the season will be above normal, but how much above normal it will be."

3. Housing: The National Association of Realtors in 2005 predicted that national median existing-home prices are "expected to rise another 6.1 percent in 2006" and median new-home prices are expected to "grow by 7.3 percent". They further said that "the national median home price has never declined since good record keeping began in 1968." Unfortunately for sellers (and fortunately for buyers), that trend did not continue in 2006.

2. Business: In early 2006, Mad Money stock prognosticator Jim Cramer made several predictions about the business world. The headliner was that GM would file for bancruptcy. Here is a snapshot of GM's stock price over the past year. Cramer's other predictions included a Citigroup - Goldman Sachs merger, Comcast buying CBS, Rupert Murdoch buying the Wall Street Journal, and a Pfizer- BMS merger. Maybe next year.

1. War: The pre-war estimate for the cost of the war in Iraq was $50-60B, and we are now shooting past $300B. In addition, the administration is no longer willing to make budget predictions past the current fiscal year, relying instead on emergency spending bills. We'll see if Democratic control of congress has any effect on the accuracy of predictions related to the war in Iraq or if budgets will continue to be wishful thinking.

If you liked these, here are some bad predictions of historic proportions from rinkworks.com and 2spare.com:

"I think there is a world market for maybe five computers." -- Thomas Watson, chairman of IBM, 1943.
"We don't like their sound, and guitar music is on the way out." -- Decca Recording Co. rejecting the Beatles, 1962.
"We will bury you." -- Nikita Krushchev, Soviet Premier, predicting Soviet communism will win over U.S. capitalism, 1958.
"Stocks have reached what looks like a permanently high plateau." -- Irving Fisher, economics professor at Yale University, 1929.
"It will be years - not in my time - before a woman will become Prime Minister." -- Margaret Thatcher, future Prime Minister, October 26th, 1969.
"The horse is here to stay but the automobile is only a novelty, a fad." -- The president of the Michigan Savings Bank advising Henry Ford's lawyer not to invest in the Ford Motor Co., 1903.

Breast Cancer Risk Linked To Red Meat?

2006-11-18T07:36:00.000-08:00

According to a recent Washington Post article, "Younger women who regularly eat red meat appear to face an increased risk for a common form of breast cancer, according to a large, well-known Harvard study of women's health."

However, STATS.org and Steven Milloy from JunkScience.com have some interesting commentary on both the reporting and the study itself. Here are a few excerpts:

STATS.org

"Keep in mind that even if there is a causal relationship, as suggested by the study – in other words, that eating more red meat actually causes breast cancer increases – decreasing meat consumption will only halve your risk of this type of cancer if you already eat the amount of meat in the highest red meat category. Those who eat one-and-a-half red meat servings per day (every day) will reap the biggest benefits from a lifestyle change to eating fewer than 3 servings per week. How big a benefit? Since we’re not given the rate of hormonally-related breast cancer among those who eat a lot of red meat, we simply can’t say."

Junkscience.com

"Is this result meaningful? Probably not for many of the usual criticisms that attend such human population studies, including: small study size (only 52 breast cancer cases were among the group of women with the highest red meat intake); poor data quality (no one knows with any certainty how much red meat any of these women actually consumed); weak statistics (despite the apparent magnitude of the reported 100 percent increase in risk, the result borders on statistical noise); and lack of biological plausibility (despite all sorts of speculation, no one knows with any certainty what causes breast cancer)."

"The study results are internally inconsistent. For example, eating between 1-3 hamburgers per week was associated with increased risk for a hormone receptor-positive tumor, but eating more than 3 beef sandwiches per week was not. Bacon and hot dogs were not associated with risk for hormone-positive tumor, but other processed meats (sausage, salami and bologna) were. That’s even odder since hot dogs are, after all, just rolled up bologna."

"The researchers stated in their report that hormone receptor-positive cancers increased 15 percent from 1992 to 1998 among women in the 40- to 49-year age group – a group that partially overlaps with the 26- to 46-year age group in the study. For this claimed increase, the researchers cited a January 2003 study that appeared in the Journal of Clinical Oncology ... The study did report that the incidence of hormone-positive tumors increased by 15 percent between 1992-1998 – at least in terms of raw numbers. However, this increase was not statistically significant. In fact, the authors of the Journal of Clinical Oncology study stated quite clearly, 'the increase among 40- to 49-year olds was within the limits of chance' – meaning this apparent increase was uncertain and, essentially, a non-result."

"They further went on to state that, 'One could argue that the increase in hormone receptor-positive tumors we observed is simply a result of the increase in the number of tumors tested for hormone receptor status, as the majority of breast tumors are hormone receptor-positive.'"

Grading the Pollsters (The Numbers Guy)

2006-11-17T05:05:00.000-08:00

Interesting article by Carl Bialik in the free section of The Wall Street Journal about the accuracy of polling. Here are a few excerpts:

"A study of 2002 polls by Prof. Franklin showed that the holder of a five-point lead in a poll won only 60% to 65% of the time, and polls showing the leader ahead by 10 points or more were dead wrong 10% of the time."

"Pollsters expected a rout by incumbent Bill Clinton in the 1996 presidential election, but he beat Republican Bob Dole by just 8.5 percentage points in the popular vote. Political scientist Everett Carll Ladd Jr. criticized the polls in the press, calling them worse than the 1948 fiasco, and prompting veteran pollster Warren Mitofsky to analyze the results."

"Mr. Mitofsky found that "no standard metric for gauging poll accuracy had been adopted by the polling community." So he went back to the SSRC's work after the 1948 election, and found another method he preferred: comparing the predicted victory margin to the actual margin, or the error on the margin. If a poll finds Candidate A ahead by three points and he wins by 10 points, that's an error on the margin of seven points. If Candidate B wins by five points, that's an error of eight points. There's no penalty for picking the wrong winner; pollsters are graded only on their error margin."

"The performance of Zogby Interactive, the unit that conducts surveys online, demonstrates the dubious value of judging polls only by whether they pick winners correctly. As Zogby noted in a press release, its online polls identified 18 of 19 Senate winners correctly. But its predictions missed by an average of 8.6 percentage points in those polls -- at least twice the average miss of four other polling operations I examined. Zogby predicted a nine-point win for Democrat Herb Kohl in Wisconsin; he won by 37 points. Democrat Maria Cantwell was expected to win by four points in Washington; she won by 17. (Zogby cooperated with WSJ.com on an online polling project that tracked some Senate and gubernatorial races.)"

Life for the next generation will be worse?

2006-11-11T21:30:00.000-08:00

A recent op-ed piece in the Washington Post by Harold Meyerson had an interesting statement at the end. The article was about how the new Democratic congress may limit free trade, but arguments like this one are used frequently to support almost any position:

"Exit polling made clear that the fair-trade Democrats have tapped into a profound national anxiety. When asked whether life for the next generation would be better, worse or about the same as life today, 40 percent responded "worse," while just 30 percent answered "better." That's a stunning figure to emerge from what has historically been perhaps the most optimistic of nations."

Is this really a stunning figure? A closer look says no. The number of people saying that their children's generation will be worse than their own is pretty consistently higher than the number saying it will be better off. Back in 1990, a NY Times poll, 36% said worse, and 28% said better. Pew data shows similarly pessimistic outlooks historically, though it varies considerably over time.

The interesting thing is that while people are pessimistic about their children's generation, they are much more optimistic about their children. For example, the 1996 Pew survey that showed only 38% of people saying their children's generation would be better off also showed 51% thought their own children would be better off. In addition, when people are asked about how their own standard of living is compared to their parents, they consistently say that they are better off.

I have two theories on these conflicting results. First, people are bad at seeing long term trends. Since our lives are relatively short, it's easy to get caught up in day-to-day changes but miss the long term trends. The history of humans on earth has shown consistent progress on everything we can measure (average lifespan, rates of illness, caloric intake, etc.), but it's easy to forget that if you just lost your job. Second, people have an unrealistically rosy view of the past. It's easy to remember your idyllic childhood without thinking about the fact that there was state-sponsored segregation in the US until the 1960s.

Factchek.org reviews the worst eview the worst deceptions from this campaign season

2006-11-04T16:08:00.000-08:00

Interesting article from factchek.org about the "unprecedented barrage of advertising containing much that is false or misleading." Here are a few excerpts:

In Ohio, a Democratic ad said Republican House candidate Joy Padgett was investigated "for abusing her position to help her own business." The truth is the investigation was triggered by an anonymous accusation and the investigators concluded there was "no substance to the allegation."

Nebraska Republican Senate candidate Pete Ricketts used newspaper headlines in one of his TV ads, but the truth is the headlines were faked. A spokesman explained this was due to "creative reasons."

A Democratic-leaning group ran false ads accusing a few Republican senators of voting to deny modern body armor for troops in Iraq. In fact, the amendment cited by the ad didn't mention body armor, and passing it wouldn't have allowed the Pentagon to acquire a single additional armored vest: It already was buying as many as the economy could produce.

Numerous Republican ads claimed Democrats wanted to "give Social Security benefits to illegal immigrants." But nobody's proposing paying a dollar of benefits to anyone while they are illegal. The ads mischaracterize Democratic support for current law, which allows immigrants to get credit for the Social Security taxes they paid while working illegally, but only if and when they become legal or gain citizenship and then become eligible to receive benefits.

U.S. Lagging on Infant Mortality?

2006-11-03T18:56:00.000-08:00

Interesting article from USNews.com about the fact that the US often does poorly in rankings of infant mortality rates across countries. A recent report by the Commonwealth Fund says that the US "infant mortality rate is 7.0 deaths per 1,000 live births, compared with 2.7 in the top three countries." And this (admittedly funny) clip from Bill Maher mentions this type of statistic as well.

However, this seemingly simple comparison across countries as not as simple as one may think. Here are a few excerpts from the article:

"First, it's shaky ground to compare U.S. infant mortality with reports from other countries. The United States counts all births as live if they show any sign of life, regardless of prematurity or size. This includes what many other countries report as stillbirths. In Austria and Germany, fetal weight must be at least 500 grams (1 pound) to count as a live birth; in other parts of Europe, such as Switzerland, the fetus must be at least 30 centimeters (12 inches) long. In Belgium and France, births at less than 26 weeks of pregnancy are registered as lifeless. And some countries don't reliably register babies who die within the first 24 hours of birth. Thus, the United States is sure to report higher infant mortality rates."

"As Nicholas Eberstadt, a scholar at the American Enterprise Institute, points out, Norway, which has one of the lowest infant mortality rates, shows no better infant survival than the United States when you factor in weight at birth."

Map of dishonest and misleading campaign ads

2006-10-28T16:39:00.000-07:00

A recent factcheck.org article said that "both political parties are functioning in the 2006 House races as factories for attack ads." Their analysis showed that 91% of NRCC ads and 81% of DCCC ads were negative as of 10/23. Even worse, many of these ads were highly misleading or had factual inaccuracies. The article has some great examples.

Unfortunately, these types of ads have become the norm. The map below uses data from factcheck.org to document some of the dishonest campaign ads this year.

A few notes:
- The map is not comprehensive, and relies solely on specific ads mentioned on factcheck.org since June of 2006.
- Note that red flags represent Republican ads, blue flags represent Democrat ads, and purple flags represent places where both Republicans and Democrats are using misleading attack ads. The point here is not to count red flags and blue flags (though factcheck.org did some analysis to that end), but to point out instances of intellectual dishonesty across party lines.
- If you see any mistakes in the map, let me know and I will change it as soon as possible.

In a Sea of Uncertainty, We All Have an Anchor

2006-10-28T16:31:00.000-07:00

Interesting article in the Washington Post about how anchoring on previous Iraqi casualty estimates may have led to skepticism about the Lancet study. A few excerpts:

"More than three decades ago, two psychologists conducted an experiment that was equal parts funny and deadly serious. They spun a roulette wheel and when it landed on the number 10 they asked some people whether the number of African countries was greater or less than 10 percent of the United Nations. Most people guessed that estimate was too low. Maybe the right answer was 25 percent, they guessed.

The psychologists spun their roulette wheel a second time and when it landed on the number 65, they asked a second group whether African countries made up 65 percent of the United Nations. That figure was too high, everyone agreed. Maybe the correct answer was 45 percent.

The difference in the estimates of the two groups was tied to the original number they were given. It made no difference that the number was meaningless: It came from a roulette wheel. Psychologists Amos Tversky and Daniel Kahneman described the error as caused by a phenomenon known as anchoring -- when you don't know the answer to something, whatever starting point you have plays a powerful role in determining what you think is the right answer."

"Previous estimates had put the number of Iraqi casualties at 30,000 to 50,000. Once that number was anchored in people's minds, it was a foregone conclusion that most people would find it very difficult to accept a much larger number.

"It could be malicious and deliberate or innocent and just wrong, but the fact that the administration had set an anchor is what makes the new number seem implausible," said Max Bazerman, who studies human decision-making at Harvard Business School.

It is important to remember that the psychological phenomenon does not tell you what the correct number of casualties in Iraq really is. But it does say that even if the 650,000 number is accurate, we are likely not to believe it."

An Unwelcome Discovery

2006-10-24T06:26:00.000-07:00

An interesting article from the New York Times about Eric Poehlman, who a year earlier had plead guilty to "lying on a federal grant application and admitted to fabricating more than a decade’s worth of scientific data on obesity, menopause and aging, much of it while conducting clinical research as a tenured faculty member at the University of Vermont." Poehlman is only the second scientist in US history to face prosecution for falsifying research data.

False Attribution: Missouri Senate Campaign Ads

2006-10-22T15:09:00.000-07:00

Check out this post from Fallacy Files on an old political campaign trick. I also recommend you check out www.factcheck.org, which catalogues these types of fallacious ads, on both sides of aisle.

The Science of Counting the Dead (STATS.org)

2006-10-22T07:37:00.000-07:00

More on the Lancet study from STATS.org. A couple excerpts:

"War-torn countries do not have central registries to record deaths. People do not necessarily die in hospitals, and their bodies are not necessarily sent to morgues. While the press makes no claim to having actually seen all the deaths that occur, the website Iraq Body Count (IBC) keeps a database of “media-reported civilian deaths in Iraq that have resulted from the 2003 military intervention by the USA and its allies.” The IBC does not count excess deaths due to a deterioration of infrastructure, lack of hospitals or clean water. Nor does it count deaths not reported by the media. At least in theory, innumerable deaths occur quietly, under the radar screen of any accounting office."

"Flares into Darkness argues that the sampling method would invariably favor densely populated areas, and that these areas would have disproportionate levels of bombs. It is certainly true that densely populated areas are more likely to be sampled – but only proportional to their population. In other words, if ten times as many people live in Region A than live in rural Region B, then Region A is ten times as likely to be chosen as a sampling destination. Overall, this will not have the effect of oversampling cities; it will have the effect of sampling cities proportional to their population, and rural areas proportional to theirs."

For those who want more, here is an interesting Scienceblog post on the subject.

Study Shows Good With Math Means Bad With People?

2006-10-20T18:50:00.000-07:00

As reported in the "science" section of Slashdot, "A recent study by Brookings Institution's Brown Center shows that students who are good with math are less likely to be happy, and are likely to have low confidence. From the article "In essence, happiness is overrated" says study author Tom Loveless. I wonder if Loveless is just a nickname, because he is so good with math."

This is a pretty unique interpretation of a study, which basically says that, across countries, those countries who's students are happiest and most confident with their math abilities do worse on standardized math tests. As reported by CNN, "Countries reporting higher levels of enjoyment and confidence among math students don't do as well in the subject."

The full context of the Loveless quote was "In essence, happiness is overrated, says study author Tom Loveless. "We might want to focus on the math that kids are learning and just be a little less obsessed with the fact that they have to enjoy every minute of it.""

Giving the slashdot author the benefit of the doubt, I will assume he was joking with his backward interpretation of the study's results. However, there were 470 comments discussing the post. Here are a few of the best, typos and all...

"Didn't anyone stop to think that maybe math is overrated?"

"Children (especially girls) who show aptitude at math are treated as if they are social misfits, and their social missteps are toerated more than in "normal" children. Kids who are good at maath are frequently "taught" via positive reinforcement to be social misfits by society."

"That's because they've found a place filled with people who think just like they do in the important ways. Perhaps future studies can show that people good with math surrounded by those who aren't will be unhappy ;)"

"It's probably because people who are bad at math are more likely to think illogically. If one is not bound by logic, one tends to ignore the odds (by being completely ignorant of them) and think more positively, relying on "luck" and "it could happen". I would imagine one would be happier always thinking there is a chance than thinking the odds are always against them."

Just a few examples of how people can take any study's findings and twist it's interpretation to suit their preconcieved notions.

Update: for those who don't see why the slashdot author's interpretation is incorrect...
1) The study refers to happiness with math, not happiness in life as the slashdot post implies.
2) The study implies that overconfidence and happiness with math cause poor test scores, while the slashdot post misinterprets this to mean that high test scores cause low happiness.
3) The study only talks about differences across countries, not differences within them. Within a country it is likely that the most confident and happy math students do better on tests.

Inconvenient Truths for Al Gore

2006-10-18T17:50:00.000-07:00

This is an old article, but I hadn't read it before. It was written by Bjorn Lomborg, author of The Skeptical Environmentalist. While he agrees with Gore that global warming is real and cause by humans, he disagrees with some of his other conclusions, and points out some bad analysis from the movie. Here are a few excerpts:

"Gore shows that glaciers have receded for 50 years. But he doesn’t acknowledge they have been shrinking since the Napoleonic wars in the early 1800’s – long before industrial CO2 emissions. Likewise, he considers Antarctica the canary in the coalmine, but again doesn’t tell the full story. He presents pictures from the 2% of Antarctica that is dramatically warming, while ignoring the 98% that has largely cooled over the past 35 years. The UN climate panel estimates that Antarctica’s snow mass will actually increase during this century. And, whereas Gore points to shrinking sea ice in the Northern Hemisphere, he fails to mention that ice in the Southern Hemisphere is increasing."

"Europe’s deadly heat waves in 2003 lead Gore to conclude that climate change will mean more fatalities. But global warming would mean fewer deaths caused by cold temperatures, which in most of the developed world vastly outweigh deaths caused by heat. In the UK alone, it is estimated that the temperature increase would cause 2,000 extra heat deaths by 2050, but result in 20,000 fewer cold deaths."

"Financial losses from weather events have increased dramatically over the past 45 years, which Gore attributes to global warming. But all or almost all of this increase comes from more people with more possessions living closer to harm’s way. If all hurricanes had hit the US with today’s demographics, the biggest damage would have been caused not by Katrina, but by a hurricane in 1926. Allowing for changes in the number of people and their wealth, flood losses have actually decreased slightly."

"The movie invites viewers to conclude that global warming caused Hurricane Katrina, with Gore claiming that the warm Caribbean waters made the storm stronger. But when Katrina made landfall, it was not a catastrophic Category 5 hurricane; it was a milder Category 3. In fact, there is no scientific consensus that global warming makes hurricanes more destructive, as he claims. The author that Gore himself relies on says that it would be “absurd to attribute the Katrina disaster to global warming.”

The Human Cost of the War in Iraq

2006-10-15T08:50:00.000-07:00

A recent study, published by The Lancet, a British medical journal, showed that approximately 600,000 people have been killed in the violence of the war that began with the U.S. invasion in March 2003." The study was conducted by the Bloomberg School of Public Health at Johns Hopkins University, the School of Medicine at Al Mustansiriya University in Baghdad, and the Center for International Studies at M.I.T.

As reported by Reuters, Tony Blair's office said "the ... figure is an order of magnitude higher than any other figure. It's not one we believe to be anywhere near accurate and that is not in any way to downplay the seriousness of the security situation in Iraq."

As reported by Will Dunham, George W. Bush stated that "I don't consider it a credible report. Neither does General (George) Casey (top U.S. commander in Iraq) and neither do Iraqi officials."

As reported by Democracy Now’s Amy Goodman, General Casey, when asked about the study stated:

GEN. GEORGE CASEY: I have not seen the study. That 650,000 number seems way, way beyond any number that I have seen. I’ve not seen a number higher than 50,000. And so, I don’t give that much credibility at all.

REPORTER: What’s the 50,000 number? Where did you see that from?

GEN. GEORGE CASEY: I don't remember, but I’ve seen it over time.

REPORTER: Is it a U.S. military estimate?

GEN. GEORGE CASEY: I don't remember where I saw that. It’s either from the Iraqi government or from us, but I don’t remember precisely.

Criticisms of the Study
There seem to be 3 basic criticisms of the study:

The death counts are much higher than any other figures that have been reported, which makes them highly suspect
Many of the deaths are not due to US actions, but due to the actions of others, for example, terrorists
The sampling methods used are flawed

Survey Methodology
Before addressing these criticisms, let me provide a simplified overview of how the study was conducted. Surveyors randomly picked 50 sites in Iraq to survey people about deaths in their household over the past 4 years. Each household in Iraq had an equal chance of being chosen for one of these sites. Once a survey site was chosen, 40 adjacent households in the area were interviewed. They compared the "death rate" in these households before the invasion to the death rate after the invasion. It turned out that the death rate pre-invasion was 5.5 deaths/1,000/year while the death rate post-invasion was 13.2 deaths/1,000/year. Applying the difference between these two numbers to the population of Iraq, and multiplying by the length of the war to date, you arrive at an estimated 654,965 excess deaths in Iraq as a consequence of the war.

Death Counts Seem Too High
Obviously, just because new information contradicts old information doesn't make it incorrect. But there are two major reasons that the death estimates seem so much higher than numbers we have heard before. First, most data on deaths is collected "passively", such as looking at hospital records, morgue counts, etc. In this survey, the data was collected "actively", by going to households and asking them. According to the study, this is a major flaw in passive studies, and "other than Bosnia, we are unable to find any major historical instances where
passive surveillance methods (such as morgue and media reports) identify more than 20% of the deaths which were found through population-based survey methods."

Second, much of the news and information we get on Iraq is focused on Baghdad. However, this makes up a relatively small percentage of Iraq's population. The chart below shows that less than 25% of Iraq's population is located in Baghdad, and roughly the same number of clusters in the study were located in Baghdad.

Deaths Not Due to Coalition Actions
The figure below shows that less than a quarter of the deaths reported were attributed to coalition forces. However, the point of this study is not to assign blame to individual parties for deaths. It is to answer the question, "if the US had not invaded Iraq in the first place, how many lives would have been spared?" The effects of war are complex, and each action can have many indirect consequences. We must take responsibility for the inevitable, though possibly hard to predict, consequences of our actions. And if we didn't predict the consequences of our actions properly, we need to spend more time thinking about them the next time.

Sampling Methods Flawed
The sampling method used is called The Cluster Survey Method. This is a standard method for estimating deaths in conflict situations. One commonly cited problem with this method is that if, for instance, a bomb dropped on one of the cluster sites, all of the families in that cluster would be affected by the war to a greater degree than average, and this could inflate the estimated deaths. This is true, however, it does not affect results in any systematic way. While there is a chance that a site could have been disproportionately affected by war, there is an equal chance it was disproportionately unaffected. The only real problem is in the confidence interval of predictions, and the study takes that into account: "To compensate for this “clustering effect” (sometimes called the design effect), the number of households or persons in a cluster sample is increased over that of a simple random sample in order to provide adequate precision."

A second commonly cited problem with the survey is that surveys are subjective and can lead to misleading results. However, death certificates were produced by 92% of respondents, making this criticism pretty unreasonable.

A third commonly cited problem with the survey is that urban areas were over-represented in the study. This complaint may have some validity. Because surveyor safety was an issue (surveyors for other projects have been killed in Iraq in the past), some accomodations had to be made. This may have lead to a systematic bias towards urban locations. However, this bias is highly unlikely to change the results by "an order of magnitude," as Blair's office states.

Conclusions
Given the conditions in Iraq, this study seems to be well thought-out and executed, and sheds some light on the number of deaths for which the war in Iraq is responsible. Like any empirical analysis based on a population sample, the exact numbers of deaths are meaningless, but the fact the bottom end of the 95% confidence interval is larger than any death count that has been reported previously is pretty telling. For more reading on the study, I recommend these two articles from the Wall Street Journal.

To those who question the validity of this study, I have a basic question: Is the result of this study important? If so, we should be commissioning a new study based on an improved methodology. If not, critics should not be arguing about the survey methods but talking about how 600,000 deaths is a small price to pay for the good we are doing. My guess is that there is no number large enough that would give supporters of the war pause as to whether we did the right thing. To be fair, there is likely no number small enough that it would slow down the war's opponents.

I think this passage from the New York Times article Faith, Certainty and the Presidency of George W. Bush sums it up well:

"The [Bush senior] aide said that guys like me were ''in what we call the reality-based community,'' which he defined as people who ''believe that solutions emerge from your judicious study of discernible reality.'' I nodded and murmured something about enlightenment principles and empiricism. He cut me off. ''That's not the way the world really works anymore,'' he continued. ''We're an empire now, and when we act, we create our own reality. And while you're studying that reality -- judiciously, as you will -- we'll act again, creating other new realities, which you can study too, and that's how things will sort out. We're history's actors . . . and you, all of you, will be left to just study what we do.''"

San Francisco Chronicle's Cosmetics Scare (STATS.org)

2006-10-11T14:53:00.000-07:00

The San Francisco Chronical recently wrote and article about how chemicals in women's cosmetics may be harmful to their health. STATS.org wrote an interesting post criticizing the article for one-sided and misleading interpretation of the facts. Here are a few excerpts from the STATS.org piece:

"It is simply untrue that the chemicals found in cosmetics have been linked to birth defects in women with “alarming levels” of these chemicals. Birth defects have been found in animals (mostly rats) that have been given extremely high doses of the chemicals. These doses are at 100-1000 times the level that humans are exposed to."

"There is little evidence to suggest that low-level exposure to humans would have the same effect as high-level exposure in animals. If every scientific discovery about rats were also true for humans, we would already have a cure for cancer."

"Just like breathing car-exhaust can’t be good for you, but we don’t ban cars because their exhaust will result in our breathing a little bit of it. Our bodies are equipped to handle a small amount of assault without developing cancer. Notably, no epidemiological studies have found that those who use more personal care products have higher rates of cancer, whereas, living in a polluted city does increase rates of cancer."

Breastfeeding kids makes them smart, right?

2006-10-08T06:12:00.000-07:00

From IQ's Corner, Joel Schneider, clinical psychologist at Illinois State University, presents a new paper which disputes the theory that breastfeeding children increases their IQ. While the study found a 4.5 point increase in IQ among breast-fed children, controlling for the mother's IQ shrinks the increase to 1.3 points. In addition, the researchers cleverly looked at a subset of mothers in their sample who breastfed some of their children but not others. The difference in scores between the siblings who were breast-fed and those who were not was not significant.

Money can buy happiness?

2006-10-08T05:39:00.000-07:00

As reported in several sources including BBC News and AOL, a study by the University of Nottingham showed that "money can buy happiness."

I couldn't find the study online, but this press release gives the basic details. The study interviewed lottery winners and non-winners as well and asked a series of both subjective and objective questions related to happiness. The results of the study that were often cited to prove the study's thesis were:

"97 per cent of interviewees were just as happy, if not happier, following their big win."
"When presented with the statement “I am satisfied with my life,” 59 per cent of the winners agreed, compared to 40 per cent [of non-winners]."
"15 per cent of winners classed themselves as single in their previous lives, this dropped to 12 per cent post-win"

While an overwhelming number of interviewees were as happy or happier after winning the lottery, we don't know whether non-winners were also happier with their lives. In the second statistic, which does compare winners with non-winners, note that they ask about "satisfaction with life", not "happiness". It is also unclear whether this new found "happiness" lasted, or diminished over time. And while the percentage of single people dropped among winners, it's not clear whether this is due to winning the lottery or to the passing of time.

In addition, I've mentioned the book Stumbling On Happiness before, which highlights that humans are not as good as we think at comparing different states of happiness. While we can be confident that we are happy at this moment, it's hard to say whether we are happier than we were 10 minutes, 10 days ago, or 10 years ago.

Finally, as usual, it is useful to note the source of the research: "The research was commissioned by National Lottery operator Camelot." If anyone sees a copy of the full study, let me know because it is possible that my concerns are addressed. But in any case, the press release does little to convince me that the study has uncovered anything noteworthy.

Fact Check it Out (Fallacy Files)

2006-10-08T05:12:00.000-07:00

From the Fallacy Files weblog, an example of congressional candidates Mike Whalen (R) and Bruce Braley (D) from Iowa using faulty reasoning to smear their opponents. Two popular methods are used: 1) quotes out of context and 2) guilt by "association". The key to avoiding these types of smear tactics is clear: 1) never utter a sentence that can't stand completely on its own if taken out of context, and 2) never do anything that a disreputable group might agree with. Or, to be safe, never say or do anything.

If you're interested, factcheck.org is full of examples of these types misleading political ads.

10 reasons people make stupid decisions

2006-10-05T08:00:00.000-07:00

Have you ever wondered why most of the people you know (and probably your elected representatives) insist on making dumb decisions when they are so clearly wrong? Below are ten reasons that they act like idiots without even knowing it. And as they say in poker, if you can't spot the sucker at the table, then it's probably you...

10. We’ve come this far… (sunk cost bias) - We all know that the past is past and we can’t get back money or time that we already spent. But many people irrationally take sunk costs, time, money, or other resources which have already been spent and can't be recovered, into their decision making. Barry Schwartz from Swarthmore College, author of The Paradox of Choice, in an LA Times op-ed piece highlights examples of sunk costs used in decision making, such as how much you spent to get your car fixed last time, how long you have been dating someone, how much you invested in a stock, or how many troops have been lost in Iraq so far.

9. Me me me! (egocentric bias) - Putting yourself in another person’s shoes is harder than it sounds for most people. Consider the example from an op-ed piece in the NY Times by Daniel Gilbert from Harvard, author of Stumbling on Happiness. In a study conducted by Sukhwinder Shergill and colleagues at University College London, pairs of volunteers were connected to a device that allowed each of them to exert pressure on the other volunteer’s fingers. The researcher began by exerting a fixed amount of pressure on the first volunteer’s finger. The first volunteer was then asked to exert the same amount of pressure on the second volunteer’s finger. The second volunteer was then asked to exert the same amount of pressure on the first volunteer’s finger, and so on. Although volunteers tried to respond with equal force, they typically responded with about 40 percent more force than they had just experienced. Each time a volunteer was touched, he touched back harder, which led the other volunteer to touch back even harder. Is this why parties in a conflict invariably think they are both "right"?

8. That just proves my point. (confirmation bias) - Isn't it a coincidence that no matter what happens in the world, politicians can spin it to show why that confirms their opinions? A cynical explanation is that politicians twist the truth to get what they want. But a more subtle explanation is that our brains tend to search for and interpret information in ways that support our pre-exisiting opinions. As explained in a Scientific American article by Michael Shermer, publisher of Skeptic, a study before the 2004 presidential election had 30 participants view statements by Kerry and Bush in which both men clearly contradicted themselves. The 15 of the participants who were strong Republicans were critical of Kerry but let Bush off the hook, and vice-versa. In addition, neuroimaging results showed that the part of the brain most associated with reasoning, the dorsolateral prefrontal cortex, was inactive. Most active were the orbital frontal cortex (processing of emotions), the anterior cingulate (conflict resolution), the posterior cingulate (making judgments about moral accountability), and the ventral striatum (reward and pleasure).

7. That’s easy. (overconfidence) - According to a famous survey of drivers conducted by Ole Svenson in 1981, 80% of respondents rated themselves in the top 30% of all drivers. And anyone who doesn't see that as a strange finding probably rated themselves in the top 30% in math also. While overconfidence is definitely a good thing in many situations, it probably means people don't work on their weaknesses as much as they should. Do overconfident drivers practice safe driving given their lack of abilities? Do overconfident doctors, discussed in this HealthDay article, get the proper training? Do overconfident public officials realize when they are making bad foreign policy decisions?

6. I’m #1! I’m #1! (dysfunctional competition) - Have mixed feelings when you find out your co-workers get a promotion that you weren't even interested in? You're not alone. People's happiness is often a function of what they have relative to others. Your co-workers might be just as happy with you getting a pay cut as they would with getting a raise themselves. In a Q&A with Max Bazerman from Harvard, "When I ask people whether they would prefer a) $7 for themselves and for another person or b) $8 for themselves and $10 for the other person, people choose "b." However, when people are simply given "a" or "b," "a" makes them happier."

5. Mine mine mine! (endowment effect) - Why is it so hard for people to throw, give away or sell things that are past their prime? One theory is that people tend to place a higher value on objects they own relative to objects they do not. As explained on behaviouralfinance.net, an experiment by Kahneman, Knetsch and Thaler found that randomly assigned owners of a mug needed to be paid around $7 for it, while randomly assigned buyers were only willing to pay around $3. So it's likely that your neighbor values things they already have more than they would if they didn't already own them. If you have ever tried executing a trade in fantasy football right after the draft, you understand this irrationality well.

4. Watch out for sharks. (availability bias) - People are suckers for recent and memorable events. So much so, that they think these types of events are more likely to happen than they actually are. An article in Wired Magazine by Ryan Singel ranked the odds of someone dying over the past 10 years from various causes, from terrorism to a hernia to accidental poisoning. Despite the focus that terrorism has gotten in the past several years, the risk of dying from terrorism is very low compared to much more mundane ways to die such as driving off the road (80 times more likely to kill you) or even a hernia (5 times more likely to kill you). A Motley Fool article even explains how misleading vividness can cause major errors in investing (all my friends have a Treo .... I should invest in Palm!). How much of a problem is this? Well, if the world has limited resources to invest, where do you think people are going to want to spend it? Stopping the slow and steady march of global warming that won't affect any of us during our lifetimes? Or more graphic threats like terrorism?

3. If everybody else think so… (conformity) - People make decisions based on what they think and not what everyone else thinks, right? Well, a famous experiment by Solomon Asch, explained on Answers.com, had one participant and several experimenters in disguise were asked to announce their judgement about the length of several lines (such as which line was longer then the other, etc.) The experimenters in disguise were instructed to give incorrect answers to these easy questions. Surprisingly, a third of respondents gave wrong answers because of the pressure of their peers. Are these types of pressures enough to get people to do things that they know are wrong (shoplifting, accounting fraud, genocide...)?

2. Lets go hard 8! (illusion of control) - We all know that there’s no difference between my chances in craps if I have the dice or someone else at the table does, right? As explained in Wikipedia, Ellen Langer from Harvard famously showed that people rolling the dice in craps threw harder on average when going for high numbers and softer when going for low numbers. And they tended to bet more when they were rolling relative to others. This may be one reason that people know gambling doesn't pay on average, yet gladly gamble their own money. It should also cause you to question the confidence of your co-workers who know they will hit an aggressive deadline even though there are many factors out of their control.

1. He's just a moron. (attribution error) - Is the driver that cut you off a jerk? Or is he a good guy who didn't see you because he's distracted by something else going on in his life. Judging by the finger that you chose to hold up, you think he's a jerk. That's about par for the course, as most people tend to over-emphasize personality-based explanations for the actions of others. A NewYorker article by James Surowiecki discusses a classic experiement where "subjects shown a person shooting a basketball in a gym with poor lighting and another person shooting a basketball in a gym with excellent lighting assume that the second person hit more shots because he was a better player." So maybe it's better to withhold judgement about a person until you've actually talked to them. Even if they did cut in front of you in line at McDonalds.

Broken Window Fallacy, On Steroids

2006-10-03T08:40:00.000-07:00

Interesting post from Coyote Blog about the recent Business Week cover story on health care called "What's Really Propping Up The Economy". He points out a flaw in the logic of the story, which suggests that inefficient health care spending is creating jobs and "propping up the economy". Here's a quote from the article:

"For years, everyone from politicians on both sides of the aisle to corporate execs to your Aunt Tilly have justifiably bemoaned American health care -- the out-of-control costs, the vast inefficiencies, the lack of access, and the often inexplicable blunders. But the very real problems with the health-care system mask a simple fact: Without it the nation's labor market would be in a deep coma. Since 2001, 1.7 million new jobs have been added in the health-care sector, which includes related industries such as pharmaceuticals and health insurance. Meanwhile, the number of private-sector jobs outside of health care is no higher than it was five years ago."

The basic flaw in this logic is a lack of accounting for opportunity costs, highlighted in the parable of the broken window. The parable basically explains that, even though you breaking your window provides work for the glazier, which means the glazier has money to spend with the baker, and so on, this doesn't mean we should go around breaking windows to support the economy. What is missing from the broken window analysis is opportunity costs, or what you would have spent the money on if you hadn't broken window.

If we weren't so inefficient in health care, it's likely that people would spend their money on other things, and we wouldn't need health care spending to "prop up the economy".

Want to make more money? Drink more.

2006-10-02T06:35:00.000-07:00

Great post from the Mutual Improvement blog. It criticizes some bad analysis discussed in this Yahoo News article. Edward Stringham, an economics professor at San Jose State University and co-author of the study with fellow researcher Bethany Peters, argue that "Social drinking builds social capital," and that
they "are out networking, building relationships, and adding contacts to their BlackBerries that result in bigger paychecks."

It points out the major flaws in the study, which is correlation implying causation. There are a number of reasons drinking and earnings might be correlated other than drinking causing earnings:

Earnings could cause drinking: Those who earn more may have more income available to spend on alcohol. This seems unlikely, but a more likely explanation is that...
A third variable could cause both drinking and earnings: While the study did control for a number of demographic variables, there are a number of important variables not included:

Occupation: Certain occupations lead to more social drinking. For instance, investment bankers earn a lot and also tend to drink more. What would be more interesting would be a study on whether investment bankers who drink earn more than investment bankers who do not.
Social Interactions: Drinking is just one of many possible ways to interact with your co-workers. It is possible that social interactions in general increase earnings, not drinking specifically. For instance, it is possible that going to lunch with your co-workers is more effective than going to the bar after work.

Here is my theory on the subject, which while not substantiated by any data of my own, would contradict the findings of the study, and is not disproved by the study data:

A given person drinking more will not increase their earnings.
However, as people gain jobs with more responsibility, social interactions with your co-workers becomes more important to doing your job effectively.
One way people have found to increase social interactions with their co-workers, is to do so while drinking alcohol, which they happen to enjoy anyway (similar to, for instance, golf).

This would say nothing about drinking, only that interacting socially with your co-workers is a good idea, especially after you get that promotion with more responsibility. Not quite as attention grabbing as "No Booze? You May Lose," which is the title of the study.

It's interesting to look at the source of this study. The study was sponsored by The Reason Foundation, whose mission is to "advance a free society by developing, applying, and promoting libertarian principles, including individual liberty, free markets, and the rule of law. We use journalism and public policy research to influence the frameworks and actions of policymakers, journalists, and opinion leaders." And since people acting freely choose to drink alcohol anyway, it's no coincidence they found data that proves that this is a good idea. While I certainly agree with many of their libertarian ideals, I think that this study does a disservice to their cause.

How Reliable Is the Zogby-Journal Poll?

2006-09-28T07:58:00.000-07:00

Not very, according to this Columbia Journalism Review article. According to Cliff Zukin, a political science professor and polling expert at Rutgers University:

"Online, Internet, opt-in polling, where people volunteer to be respondents, doesn't really have a basis in scientific validity. There are two kinds of samples in the world. There are probability samples, and there are non-probability samples."

The Zogby interactive polls, says Zukin, clearly fall into the latter camp. "With probability samples, when everybody has a known chance of being selected, you can make pretty valid inferences about the population from which it is drawn," says Zukin. "You can't do that at all with self-selected surveys. That's a problem."

Another problem with Internet-based polling, says Zukin, is that, in general, Web and email-based surveys tend to overvalue the opinions of young people. A group that is notoriously lousy at showing up to actually vote.

"Internet coverage is now about two-thirds of the population," says Zukin. "But it's really age-skewed and, to a lesser extent, education-skewed, in the wrong way for voters. It's younger people who are online. It's older people who are not online. It's older people who vote. And younger people who don't."

Can political science literatures be believed?

2006-09-27T06:25:00.000-07:00

From The Washington Monthly, a paper was recently published by Alan Gerber (Yale) and Neil Malhotra (Stanford) about "publication bias, the tendency of academic journals to publish studies that find positive results but not to publish studies that fail to find results."

The basic hypothesis is that papers that do not show statistically significant results are less likely to be published than those that do (though this may or may not be a problem). However, this bias may lead researchers to do anything they can to produce statistically significant results. If that is the case, you would expect many papers to show significance that barely meets the 95% confidence threshold.

That's just what the paper shows (see the figure below). A z-score of 1.96 on the x-axis is the threshold for statistical significance, and you can see a surprising number of papers at that level.

Here's what the authors say:

"It is sometimes suggested that insignificant findings end up in “file drawers,” but we observe many results with z-statistics between zero and the critical value. There is, however, no way to know how many studies are “missing.” If scholars “tweak” regression specifications and samples to move barely insignificant results above conventional thresholds, then there may be many z-statistics below the critical value, but an inordinate number barely above the critical value and not very many barely below it. We see that pattern in the data."

Just An Online Minute… Stevens’ Flawed Poll

2006-09-25T09:16:00.000-07:00

Interesting article from the Just And Online Minute blog. It talks about a poll highlighted by Republican Senator Ted Stevens, paid for by Verizon, and conducted by Public Opinion Strategies and The Glover Park Group. The poll shows that the public disapproves of net neutrality legislation. I'll let you read the poll question for yourself and judge whether it was well written:

“Which of the following two items do you think is the most important to you:

Delivering the benefits of new TV and video choice so consumers will see increased competition and lower prices for cable TV,

Enhancing Internet neutrality by barring high speed internet providers from offering specialized services like faster speed and increased security for a fee?"

Poor-Mouthing Prosperity: Does today's free market create too much insecurity?

2006-09-21T06:52:00.000-07:00

Interesting article from The Wall Street Journal by Brink Lindsey, Vice President for Research at the Cato Institute, and author of "Again the Dead Hand: The Uncertain Struggle for Global Capitalism". He critiques arguments made by Jacob Hacker, a Yale professor and author of The Great Risk Shift.

The argument being put forward by Mr. Hacker are that there were negative consequences to "America's sweeping transformation away from an all-in-the-same-boat philosophy of shared risk toward a go-it-alone vision of personal responsibility." Namely, he states that fluctuations in family income are much greater than they were in the 1970s.

However, as pointed out by Mr. Lindsey, he fails to take into account how people's savings have buffered this risk. "Research by economists Dirk Krueger and Fabrizio Perri has shown that big increases in the dispersion of income have not translated into equivalent increases in consumption inequality. In other words, most Americans are able to use savings and borrowing to maintain stable living standards even in the face of economic ups and downs. And those standards are much higher than those of the all-in-the-same-boat era."

While I strongly agree with Mr. Lindsey on this one, he makes some of the same over-simplificaiton mistakes in his use of data. He states a number of facts and figures about how things are beter off now than they were in the 1970s. "Back in 1970, during Mr. Hacker's golden age of economic stability and risk-sharing, the age-adjusted death rate stood at 12.2 deaths per 1,000 people. By 2002, it had fallen more than 30%, to 8.5 per 1,000."

Again, while I agree with Mr. Lindsey, my question is "so what?" Does this prove that our current form of capitalism is the cause for this? Or could there be other causes? Would that not have occured if the "welfare state" was bigger? The divorce rate in the US has also increased a great deal since 1970. Did capitalism cause that too?

This is a case of two people who believe very strongly in their ideas, and will find and interpret data any way they can to prove their point. I think it also highlights the limitations of putting any weight on single variable data like death rates over time. No matter what your cause, it's just too easy to find data like this to support it.

Overstating Border Reform's Price

2006-09-19T19:34:00.000-07:00

Interesting Washington Post op-ed piece by Robert Greenstein and James Horney from the Center on Budget and Policy Priorities. It points out some bad math in press coverage of a recent Congressional Budget Office report on the cost of the proposed immigration reform bill. Articles like this one state that the bill will cost $126B over 10 years, but fail to take into account increased tax revenue and positive effects on the economy. Whatever your feelings on immigration (read my post on some horrible arguments being put forth by a pro-immigration reform group), you have to agree with Greenstein and Horney who state "Because it is a subject that stirs strong emotions, debates over immigration policy demand fact and solid analysis".

The Duh Report (STATS.org)

2006-09-17T09:48:00.001-07:00

Interesting article from STATS.org poking holes in studies about:
- a linkage between IQ and hangovers (correlation implying causation)
- cell phone "addiction" (vague use of language)
- a linkage between celebrity and narcissism (correlation implying causation)

High school dropouts face steeper costs in U.S.?

2006-09-14T20:23:00.000-07:00

As reported by the Associated Press, a study by the Paris-based Organization for Economic Cooperation and Development shows that "dropping out of high school has its costs around the globe, but nowhere steeper than in the United States."

The articles stated that "adults who don't finish high school in the U.S. earn 65 percent of what people who have high school degrees make," while "adults without a high school diploma typically make about 80 percent of the salaries earned by high school graduates in nations across Asia, Europe and elsewhere. Countries such as Finland, Belgium, Germany and Sweden have the smallest gaps in earnings between dropouts and graduates." In addition, "44 percent of adults without high school degrees in the United States have low incomes -- that is, they make half of the country's median income or less."

Sounds like a pretty bad indictment of the US educational system, or at least evidence of classism in the US. In fact the article goes on to state "the importance of a high school degree on income ... depends on the demands for skills, the supply of workers, minimum wage laws and the strength of unions. The disparity is more pronounced in the United States ... other nations protect people with weak education qualifications through regulations or tax systems that favor the low skilled, she said ... the United States more richly rewards those who go to college."

But taking a closer look at the data, it isn't clear what the problem is. If our goal is to get as many people to pursue higher education as possible, then having big incentives to study hard is a good thing. If the US's 65% is too low, and the 80% in most other countries is better, should we be striving to create an educational system where a high-school drop-out earns the same as a Ph.D?

In addition, comparing the difference between college and non-college incomes across countries is like saying that Dwyane Wade (6' 4") is short because he is often standing next to Shaquille O'Neal (7' 1"). How do absolute non-college incomes compare between countries?

In fact, the OECD briefing note on the United States provides largely good news about American education, with some cautionary notes about how other countries are catching up. The executive summary says that "whereas in the past the US topped the league on these measures, other countries are catching up and in some cases surpassing the US performance." The goal of the OECD appears to be getting as many people to go to college as possible and not comment on "minimum wage laws and the strength of unions."

Changes in Nicotine Yield?

2006-09-13T19:43:00.000-07:00

A recent report from the Massachusetts Department of Public Health found that the amount of nicotine inhaled by the average smoker has increased by 10% between 1998 and 2004. However, Philip Morris issued a press release disputing the claims made shortly thereafter, which also got wide press coverage.

The original data from DPH showed that nicotine yield (a measure of the amount of nicotine consumed per cigarette) had increased from 1.72 in 1998 to 1.89 in 2004. According to DPH Commissioner Paul Cote, "we want health care providers to know that smokers are getting more nicotine than in the past and may need additional help in trying to quit." However, reactions to this data by the press and others pointed fingers at tobacco companies. "These reports are stunning," said Matthew L. Myers, president of the Campaign for Tobacco-Free Kids. "What's critical is the consistency of the increase, which leads to the conclusion that it has to have been conscious and deliberate."

The Philip Morris response highlighted two main concerns with the findings:

DPH did not include data from either 1997 or 2005 in their results, and if they had included that data, the upward trend in nicotine yield levels would have looked quite different.
Nicotine Yield data is derived from machine smoking tests, which are not an accurate way to determine what is actually delivered to the smoker's lungs.

Bad Analysis from DPH
There are two major problems with the DPH data. First, as stated by Philip Morris, including data from a couple more years changes the picture significantly. I have yet to read an explanation from DPH as to why those two years were cut out, which seriously calls into question the study. Second, in order to come up with the nicotine yield every year across cigarettes, they averaged the nicotine yield across brands and sub-brands (yes, a simple linear average). This is highly suspect. A straight average discounts differences in consumption across different brands (i.e. it would be like deciding who wins the presidential election by simply counting states instead of looking at electoral votes). Philip Morris released data on Marlboro which used sales-weighting instead of a straight average. The following data from DPH and Marlboro shows these problems with the DPH data:

A couple of comments about the data. First, the DPH data includes all cigarette brands, while the other data includes just Marlboro. That is why the red line and the blue line are not the same. You can deduce from this that the average nicotine yield for brands other than Marlboro rose more than it did for Marlboro. Second, r-squared is a pretty meaningless metric in this study, but I included it because it was cited in the DPH factsheet. If anyone has better data than this or can better explain the DPH methodology, I would love to hear it. The DPH report didn't include any of the underlying data on which the study was based.

Bad Analysis from Philip Morris
The second Philip Morris complaint about the was that a machine smoking test was used to arrive at the data (the question the methods defense). They said in their press release that "many public health authorities agree that machine test methods are not an accurate way to determine what is actually delivered to the smoker's lungs." First of all, even if the method is flawed in measuring the amount of nicotine delivered, it still may be perfectly suitable for measuring differences in nicotine delivery over time. Second, making a claim about the fact that "many public health authorities agree" is far too vaguely worded to be useful. In my reading, it seems the main problem with machine smoking tests are that it is hard to take into account individual differences in how people smoke (do they cover ventilation holes with their fingers? how deeply do they inhale?) However, in order for these problems to invalidate the DPH study, the machine smoking test would have to systematically report higher nicotine yields over time.

Summary
The DPH data certainly does not prove that cigarrette manufacturers (at least PM) deliberately manipulated nicotine levels in cigarrettes. But the real question, which is lost in debate over nicotine yield testing methods, is how highly regulated the industry should be. Instead of handing suspect data to journalists and letting them argue that someone needs to watch the evil tobacco companies, it might be more fruitful to focus on the undisputable facts that cigarrettes are addictive and have terrible health consequences. But I guess that's not news.

ABC's Untrue Path: "Docudramas" are the worst draft of history

2006-09-11T13:02:00.000-07:00

Interesting article from John Fund on the WSJ editorial page talking about how "docudramas" are a poor way to teach history and convey political opinions.

In my opinion, resorting to fiction in order to prove a point or teach a lesson means either 1) the facts and reasoning needed to prove your point don't stand up on their own, 2) you were too lazy to do the hard work needed to gather the facts (or present them in an interesting way), or 3) you think I am too stupid or intellectually lazy to want to learn the facts and their subtleties.

Poll Watch (Fallacy Files)

2006-09-06T06:43:00.000-07:00

Great post on Fallacy Files. CNN reported that "opposition among Americans to the war in Iraq has reached a new high" but didn't take into account sampling error. So their "news" is really no news.

Group Fighting Online Liquor Sales Serves Up a Questionable Stat on Kids

2006-09-05T15:19:00.000-07:00

Great article from the Numbers Guy at the WSJ. An analysis from the Wine and Spirits Wholesalers of America which said that millions of kids were buying alcohol over the internet had some serious flaws. Here are the highlights:

Bad sample set: They surveyed teenagers online, hardly a representative sample of all teenagers, especially when you consider that the type of people who would fill out an online survey are heavy internet users.
Bad logic: They used a question asking teenagers whether they knew anyone who had purchased alcohol online to estimate the number who actually had.

The kicker is that the sponsor of this research "are local distributors who compete with online liquor sellers".

Precisely False vs. Approximately Right: A Reader’s Guide to Polls

2006-09-04T18:54:00.000-07:00

Great article by Jack Rosenthal, president of The New York Times Company Foundation, and formerly a senior editor of The Times for 26 years. It lays out some of the intentional and unintentional mistakes made designing and interpreting polling data. Here is a quick summary:

False Precision - Extra decimal places don't mean much if the margin of error is 5%.
Sampling Error - If you don't have a random sample from your target population, you can't make conclusions about that population.
Questions - The wording or order of questions can make a big difference in outcomes.
Answers - People lie so as not to appear stupid or uninformed.
Intensity - Intensity of opinion can matter just as much as the percentage split in opinions.

Common Sense on Mass Immigration?

2006-08-29T06:25:00.000-07:00

The following is a critical look at some of the data and analysis used in an anti-immigration pamphlet called "Common Sense on Mass Immigration". It was published by the Social Contract Press, which is in favor of "fewer admissions in order to reduce the rate of America's population growth, protect jobs, preserve the environment, and foster assimilation."

"Nearly half of California's drunk driving arrests in 2001 were Latino men, and the state's percentage of fatal hit-and-run accidents in that year was more than twice the national average, namely 7.8 percent versus 3.8 percent nationally." - Mass Immigration and Personal Security / Street Crime
Correlation Implying Causation: Just because two statistics are correlated does not prove that one causes the other (i.e. just because California has more fatal hit and run accidents and also has more illegal immigrants doesn't mean the illegal immigrants are causing the accidents). California also accounts for 99% of artichoke production. Does this prove that illegal immigration causes artichoke production? Also, why does the author only cite California? What about other border states with high immigration populations?
Poor Explanation of Data: What exactly does "percentage of fatal hit-and-run accidents" mean anyway? Does it mean the percentage of hit-and-run accidents that are fatal? Or is it a measure of fatal hit-and-run accidents per capita? If it is the former, is the author implying that immigrants are more dangerous because when they hit and run, they are more likely to kill?
Mixing Up Issues: Note how the author starts by using data about Latino men, while the issue at hand is illegal immigrants. Obviously, not all Latino men are illegal immigrants.

"School violence is more frequently encountered in districts with high concentrations of immigrants. So is remedial education." - Mass Immigration and Education
Correlation Implying Causation: Just because school violence is correlated with high concentrations of immigrants doesn't mean immigration causes school violence. More likely is that immigrants are poor, and move to poor neighborhoods, and school violence is more prevalent in poor neighborhoods (i.e. both immigration and school violence are correlated with poverty).

"'In some areas of the country, 12% of felonies, 25% of burglaries and 34% of thefts are committed by illegal aliens,' according to the Congressional testimony of John Morganelli, a Pennsylvania District Attorney. Further, more than one-quarter of federal prisoners are illegal aliens. The percentage is similar for Los Angeles County's jail, where criminal aliens cost the county over $150 million annually." - Mass Immigration and Personal Security / Street Crime
Selectively Choosing Data Set to Prove a Point: The obvious question is "which areas of the country does this data come from"? If you are talking about Bismark, ND then this data is surprising, but if you are talking about a border town, it might not be. Again, notice that they say that one-quarter of federal prisoners are illegal aliens.
Bad Sourcing of Data: As usual, the author doesn't do his homework to find out where the underlying data comes from, they just take someone's word for it.

"Per pupil expenditure rose from $2,290 in 1980 to $8,745 in 2002. K-12 education now costs $415 billion, about four percent of Gross Domestic Product." - Mass Immigration and Education
Providing Data Out of Context: Wow, that increase in per-pupil expenditure sounds like a lot, right? It's actually about 6%, which is farily high but certainly doesn't sound as shocking. Also, they show what percent of GDP expenditures are today, but not what they were in 1980.
Correlation Implying Causation: Is the reason education costs have risen because of illegal immigration (implied) or because we have decided as a country that it is important enough to spend more money on?

"Because a strong middle class insures stability, saving American jobs for American citizens is as vital to our national security as the war against terrorism." - Mass Immigration: The Economy and Jobs
Mixing Up Issues: Ignoring all of the questionable assumptions in this statement (a strong middle class "insures" stability, closed borders would create a strong middle class in the long run, etc.), the biggest offense is how the author deftly segues betwen immigration and terrorism. It's interesting that the cover of the entire pamphlet is of the former Twin Towers.

"Immigrant students who go through some type of bilingual education appear to earn significantly less than their counterparts in English immersion classes." - Mass Immigration and Education
Mixing Up Issues: This is a great argument for English immersion classes but irrelevant to the argument at hand (severely limiting immigration).

"Today almost one in five Americans speaks a language other than English at home." - Mass Immigration and Education
Poor Explanation of Data: But how many of those one in five also speak English at home, or are fluent in English? I would wager that most people think that speaking a second language is a good thing, as long as English is one of them.

"In reality, any plan that allows illegal aliens to get legal residence and a green card is an amnesty. It is sufficient enticement for millions more aliens to enter or remain in this country illegally, even if it means risking their own lives and the lives of their families to get here. Another “jackpot” loophole needs to be closed: Each year we hold a lottery for 50,000 visas for legal immigration. This nonsensical approach to selecting newcomers should be ended. Amnesty may be the jackpot for illegal aliens, but it’s a losing policy for America." - Mass Immigration and Amnesty: Winning the Jackpot for Breaking the Law
Mixing Up Issues: While amnesty certainly is a benefit to illegal aliens, the visa lottery is a legal way for immigrants to enter the country. Sandwiching this statement between two paragraphs about illegal immigration makes the visa lottery seem illegal.

“ALL nineteen of the 9/11 terrorists were given entry visas by the State Department despite laws that barred all of them.” - Mass Immigration and National Security
Mixing Up Issues: If this is the case, then we should better enforce the laws already on the books, not pass new ones that stop (one would assume) hard working and well-meaning immigrants from entering the country.

Bad Analysis

2006-08-28T14:06:00.000-07:00

The purpose of this blog is to uncover bad analysis. There is no political point of view, other than my belief that the best way to make decisions is through fact-based, thoughtful, and honest analysis. Unfortunately, this isn't always easy to do, and the results don't always agree with our initial opinions. My goal is to uncover those among us who don't like doing this hard work and want to take intellectual short-cuts.