[EL] Uh oh, Rick...

Sat Oct 25 08:04:50 PDT 2014

Fair enough, except that is not how the analysis is presented. There is no discussion of any potential biases in the alternative definitions of noncitizen voters. Everyone who is matched or say they voted are included in the analysis, and then the authors plow ahead.  Indeed, the “upward bound” seems to be the preferred measure when assessing overall effects, with no discussion of a lower bound, which might be from the authors’ perspective the 5 noncitizens who said the voted and were vote validated (with the caveat about respondents’ errors and matching errors). It should be obvious to even people without a statistics background you can’t reliably statistically analyze 5 of anything.

If the errors of respondents incorrectly answering the citizenship or voting questions or being falsely matched as voted are random, then that would also explain how noncitizen voters and nonvoters are statistically indistinguishable (which is the authors’ Appendix argument that their scoring of noncitizen voters is correct). Still, if the tests show noncitizen nonvoters and voters are statistically different, that would undercut the authors’ argument without their framework. So, I tried doing the Appendix statistical tests myself from the data presented in the paper last night. I needed some numbers reported in the main body of the paper and I came to the realization that the numbers of noncitizen voters are inconsistently reported in the paper. To be fair, this could be a consequence of missing data for a demographic category or issue area. Still, we don’t usually see high rates of missing data on race and ethnicity, so it seems strange to me that Table 1 would report 2008 noncitizen voters equal to 84 and Table A.2 would have only 48 (those numbers are not transposed by me). Replication is needed to uncover what is going on here.

A reason why the tests cannot distinguish noncitizen nonvoters and voters is that there are so few observations that the “margin of error” on some subgroups is so large that statistical tests will have difficulty distinguishing between the characteristics of noncitizen voters and nonvoters. 

There is also further apparent cherry picking of data, for example, the authors analyze the demographic characteristics of noncitizen voters and nonvoters on the 2008 CCES but not the 2010 CCES. Was the 2010 analysis inconsistent with the authors’ argument, and thus excluded? Again, replication will tell. It seems obvious both years should have been reported, perhaps even pooled together, especially considering the small number of observations.

============
Dr. Michael P. McDonald
Associate Professor
University of Florida
Department of Political Science
234 Anderson Hall
P.O. Box 117325
Gainesville, FL 32611

phone:   352-273-2371 (office)
e-mail:  dr.michael.p.mcdonald at gmail.com                
web:      <http://www.electproject.org/> www.ElectProject.org 
twitter: @ElectProject

From: Jonathan Adler [mailto:jha5 at case.edu] 
Sent: Saturday, October 25, 2014 10:27 AM
To: Michael McDonald
Cc: law-election at UCI.edu
Subject: Re: [EL] Uh oh, Rick...

I have not focused on this study, but wouldn't one reason to include both categories 2 and 3 be to develop an upper-bound, with the understanding that the actual number of non-citizen voters is almost certainly lower?

Just a thought.

On Fri, Oct 24, 2014 at 9:46 PM, Michael McDonald <dr.michael.p.mcdonald at gmail.com <mailto:dr.michael.p.mcdonald at gmail.com> > wrote:
Some more thoughts:

The authors really want to have it both ways in expanding the number of noncitizen voters/registrants. There are three types of matches they use in their definition:

1. noncitizen respondents who say they are registered and have a voter file match
2. noncitizen respondents who say they are not registered and have a voter file match
3. noncitizen respondents who say they are registered but do not have a voter file match

#2 and #3 are at odds with one another. In #2, the authors assume the respondents are not truthful (for whatever reason) and the voter file is accurate. In #3, the authors assume the respondents are truthful but the matching procedure is flawed. You really can’t have it both ways. Either you trust the respondents or the matching procedure. If you think there are errors in self-reports and/or the matching procedure, then it is important to quantify the bias and magnitude of the errors.

It also strikes me that both the 2008 and 2010 surveys have registered voters as the sample frame. In 2008, 11 noncitizens who said they were registered were matched to the voter file (category #1). In 2010, even though the sample size increased by 22,800 respondents, there were zero respondents in category #1. Why did the number do down to zero? Did the matching algorithm improve so that there were fewer false positives of noncitizens matched to the voter file? Also note the combined #2 and #3 declined from 67 (19.8% of self-reported noncitizens) in 2008 to 76 (15.6%) in 2010. To the first, we might add the 11 in category #1 since they are in the authors’ universe. It seems like the change would be outside the sampling error, and might be due to improved matching algorithms finding fewer false positives.

One might say that election administration improved between 2008 and 2010 to knock more noncitizens off the voter rolls, which explains why there were fewer noncitizen registrants in 2010 compared to 2008. There is not a lot of evidence of that. Story after story of allegations of massive noncitizen voting end in a whimper when errors are found in the matching algorithms that uncovered the alleged noncitizen voting. It stands to reason if election officials have trouble with matching their voter files to identify noncitizens, Catalist might have similar problems when matching people who fit the profile of noncitizens (e.g., Latino names that are matched between the CCES and the voter file using fuzzy logic). 

I want to correct something I wrote previously, there are 6 issue areas, not 8, analyzed in Appendix Table a3 (actually five since the authors drop one of the areas without comment). Since Table 3 reports responses for all noncitizens, and no noncitizen voters were in three of the issue areas, I can fill in the missing issue areas in Table a4

Fine Businesses
Noncitizen non-voters 35.3%
Noncitizen voters 0%
Increase guest workers
Noncitizen non-voters 47.1%
Noncitizen voters 0%

If one were to run a Chi-squared test to determine if noncitizen voters and nonvoters are different on the issues, then these areas should be included in the test. They are not.

============
Dr. Michael P. McDonald
Associate Professor
University of Florida
Department of Political Science
234 Anderson Hall
P.O. Box 117325
Gainesville, FL 32611

phone:   352-273-2371 <tel:352-273-2371>  (office)
e-mail:  dr.michael.p.mcdonald at gmail.com <mailto:dr.michael.p.mcdonald at gmail.com>                 
web:      <http://www.electproject.org/> www.ElectProject.org 
twitter: @ElectProject

From: law-election-bounces at department-lists.uci.edu <mailto:law-election-bounces at department-lists.uci.edu>  [mailto:law-election-bounces at department-lists.uci.edu <mailto:law-election-bounces at department-lists.uci.edu> ] On Behalf Of Michael McDonald
Sent: Friday, October 24, 2014 8:05 PM
To: 'law-election at UCI.edu'
Subject: Re: [EL] Uh oh, Rick...

I’d like to replicate the analyses to have a better criticism, but at first blush I am not as confident that the authors do a convincing job of showing that the people identifying in the survey as non-citizens are actually non-citizens who voted.

Those who favor voter id should welcome my critique, because on p.152 the authors claim that photo id requirements are ineffective to stop non-citizen voting.

The authors’ evidence rests on two surveys, the 2008 and 2010 CCES surveys. The sample sizes of these surveys are 32,800 and 55,400. In 2008, 11 respondents identified as non-citizens who said they registered to vote and were matched to a voter list as being registered. In 2010 there were zero respondents in this category (Table 1, p.152).  There is no match to verify if these individuals are really non-citizens or had fat thumbs when they pressed buttons on their computer, or even understood the questions (the CCES is an internet survey).  To get higher numbers, the authors either use self-reports of non-citizens who self-reported being registered and were not matched to the voter files or reported not being registered and were matched with the voter files. I have many reservations about matching procedures and (from a legal perspective) would want independent confirmation from any of these three matching types that these individuals were in fact non-citizens who were registered to vote, especially since we are talking about small numbers of matches that could be a consequence of statistical flukes or other problems with the matching process.

The authors do not report if any of the 11 noncitizens who were validated as registered in fact voted (again granting that the matching algorithm didn’t produce a false positive and that the respondents didn’t misreport their citizenship status). They instead use their larger definition to find 48 non-citizens voted and 291 did not. As a percentage of the sample, even these numbers are exceedingly small and survey researchers generally would have less confidence in such small numbers. This is where next the heavy use of weighting that Vladimir references comes in.

Why I want to replicate their findings comes from their Appendix analyses, where the authors attempt to convince the reader that the non-citizen voters are really non-citizen voters. There’s a sleight of hand at work here that strikes me as cherry picking of data. In Table A3 (p.156), the authors examine 8 issue areas to convince us that the non-citizens are different than citizens. In Table A4 (p.156) the authors attempt to show through a similar analysis that noncitizen voters are the same as noncitizen non-voters. Here, they present only 3 issue areas because there were zero non-citizen voters who answered affirmatively to 5 issues they choose not to present. They conclude on the 3 issues the non-citizen non-voters are the same as non-citizen voters, but that is obviously not true for the 5 other issue areas that they choose not to report. For this glaring cherry picking of evidence, I’m even more highly skeptical and want to do a full blown replication to see where else there may be issues with their methods.

============
Dr. Michael P. McDonald
Associate Professor
University of Florida
Department of Political Science
234 Anderson Hall
P.O. Box 117325
Gainesville, FL 32611

phone:   352-273-2371 <tel:352-273-2371>  (office)
e-mail:  dr.michael.p.mcdonald at gmail.com <mailto:dr.michael.p.mcdonald at gmail.com>                 
web:      <http://www.electproject.org/> www.ElectProject.org 
twitter: @ElectProject

From: law-election-bounces at department-lists.uci.edu <mailto:law-election-bounces at department-lists.uci.edu>  [mailto:law-election-bounces at department-lists.uci.edu] On Behalf Of Kogan, Vladimir
Sent: Friday, October 24, 2014 7:11 PM
To: Rick Hasen; Steve Hoersting
Cc: law-election at UCI.edu <mailto:law-election at UCI.edu> 
Subject: Re: [EL] Uh oh, Rick...

I would be very careful about drawing broader conclusion about the incidence of non-citizen voting based on this study. The authors do a convincing job of showing that (1) the people who identified as being noncitizens in the survey are actually noncitizens and (2) those who say they voted actually voted.

I’m less convinced that we can generalize from this sample. The data is from the Cooperative Congressional Elections Studies survey, which is based on a non-representative opt-in panel from YouGov/Polimetrix. YouGov has a methodology for making their samples look like a random sample, and they have an excellent track record of predicting actual election outcomes. But I would be much more cautious about drawing conclusions about the representativeness of this sub-sub-population. As the authors themselves note, the educational levels among the noncitizens in their sample are much higher than the average among all noncitizens in the U.S. They try to use survey weights to get at this, but this only works as well as the demographics you’re using to construct the weights. Unless you think the noncitizens who sign-up to be in the YouGov panel are representative of noncitizens who do not, I’m not sure how much this teaches us about the aggregate rate of non-citizen voting. 

_______________________________________________
Law-election mailing list
Law-election at department-lists.uci.edu <mailto:Law-election at department-lists.uci.edu> 
http://department-lists.uci.edu/mailman/listinfo/law-election

-- 
Jonathan H. Adler 
Johan Verheij Memorial Professor of Law 
Director, Center for Business Law & Regulation 
Case Western Reserve University School of Law 
11075 East Boulevard 
Cleveland, OH 44106 
ph) 216-368-2535 
fax) 216-368-2086 
cell) 202-255-3012 
 <mailto:jha5 at case.edu> jha5 at case.edu 
SSRN: http://ssrn.com/author=183995
http://www.washingtonpost.com/news/volokh-conspiracy/wp/author/adlerj/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://webshare.law.ucla.edu/Listservs/law-election/attachments/20141025/e66b83fc/attachment.html>