Having just arrived home from working the night at the exit poll service
(Edison-Mitofsky), perhaps I can shed some light on this issue. First, it
was clear to us in the mid-afternoon that the exit polls were over-sampling
women, which was producing a pro-Democratic bias in the sample. Corrective
measures were taken, and we alerted our subscribers well before the polls
closed that the exit polls were showing the bias, which delayed clear Kerry
calls in states like PA and NH. Perhaps you heard the phrase "not enough
information" throughout the night, as we were waiting for actual vote
returns to confirm the exit polls.
How the exit polls are conducted, how they are weighted, and how they are
used by the networks on election night go well beyond simple sampling
methodology. If a race is outside the margin of error AND it comports with
the best previous guess of the election outcome AND it comports with
previous vote shares for various offices in the sample precincts, then we
are able to call a race immediately when the polls close. If not, a
combination of exit poll results, past election results, and actual election
returns are used to make calls as data come in from the Associated Press.
The system worked as it should, and in no case did we make a call this
election that later had to be reversed.
The only issue was a delay in getting the calls out for a few states that
might have been called sooner if we had more confidence in the exit polls.
I think we can all agree it is more important to get it right than get it
quickly, and in some cases we held back calls even longer because we weren't
certain that something funny was going on with the data, like it did in FL
in 2000. In PA, for example, the Philly suburbs were late in reporting so
we didn't want to make a call without some hard numbers from those areas to
confirm our exit polling. For states like FL and OH this time, that were
close, there was not enough information in the exit polls to make a call in
those states when the polls closed, so we were going to have to wait for the
raw numbers came in anyway.
The real problem with the exit polls is that everybody wants to know the
results before the polls close. Once the first wave of polls went out to
the subscribers, it took about a half hour for our polls (and some fake
polls) to make it onto the internet. Warren Mitofsky continually warns
everyone not to trust the early exit polls because they are what they
are...polls, and they are subject to error (not just sampling error). Too
bad more people don't appreciate this point.
=======================
Dr. Michael P. McDonald
Visiting Fellow, Brookings Institution
Assistant Professor, George Mason University
elections.gmu.edu
mmcdon@gmu.edu
703-993-4191
-----Original Message-----
From: owner-election-law_gl@majordomo.lls.edu
[mailto:owner-election-law_gl@majordomo.lls.edu]On Behalf Of Lash LaRue
Sent: Wednesday, November 03, 2004 1:47 PM
To: VOLOKH@law.ucla.edu
Cc: election-law@majordomo.lls.edu
Subject: Re: Margins of error and sample sizes in exit polls
Eugene, ....... Exit polls are a problem in two ways, only one of
which is "purely mathematical."
The math problem is the random variation that one gets in taking
samples. If one samples any population repeatedly, the several samples
will differ among themselves in ways that are rather predictable. The
standard Gaussian bell curve generates the theory of how these
variations are patterned. Of course, one must make some assumptions
about the nature of the population that is being sampled before any of
this logic can get going, and a second "of course" is that the
population of voters does not fit these assumptions perfectly. However,
the fit is close enough and this part of the problem is mathematically
well behaved. (Perhaps I should say that "it is close enough for
government work.")
The problem that is not purely mathematical (and this problem is the
serious problem) is that exits polls do not use random samples.
Depending on where one is in the country, people vary in their
willingness to respond to the pollsters. So those who do respond are
"self-selected" and thus a non-random sample; they are not typical of
the larger population of those who voted. At the worst, which does not
always happen, one gets the equivalent of a "Literary Digest" poll.
Those who are in the business try to compensate by comparing the
results of the exit polls with the actual numbers reported from early
precincts. If the margins in the exit polls are large enough (see Sen.
Clinton's run in NY), this caution is not needed. But if things are
close, this caution is absolutely necessary. At any rate, one discovers
by this empirical comparison how to "discount" the polls. The debacle of
the Florida exit polls in 2000 was caused by a simple error in
calculation. The precinct data that was used as a benchmark to judge the
accuracy of the polls was erroneous, since addition errors were made in
reporting these early totals.
By the way, the "tilt" in exit polls differs from election to election,
region to region, in ways that are not well understood.
I went to a presentation at the American Statistical Association on
this matter after the 2000 election. One of the speakers, an employee of
Voter News Service (the group that had egg on their collective faces),
began with two power point slides that read: first slide, "If you make
your living using probability theory"; second slide, "you are taking a
chance."
I hope this answers the question, ............ Lash /
Lewis Henry LaRue
Washington and Lee University
School of Law
Lexington, VA 24450-0303
email: laruel@wlu.edu
phone: 540-458-8513
fax: 540-458-8488
"Volokh, Eugene" <VOLOKH@law.ucla.edu> 11/3/2004 12:19:42 PM >>>
What are the typical margins of error (that is to say, the
purely mathematical ones) and sample sizes in exit polls? Thanks,
Eugene