The next Workshop on Quantitative Methods in Education, Health and the Social Sciences (QMeHSS) will be on Friday February 6th from 11:00-12:30pm. The upcoming workshop will be led by Dr. Bruce Spencer from Northwestern University. The workshop will be held in NORC conference room 344. NORC is located at 1155 E. 60th Street. We look forward to seeing you there.
Should Society Try to Measure the Accuracy of Verdicts in Criminal Trials? - A Total Survey Error (TSE) Perspective
Bruce D. Spencer
Professor of Statistics, Faculty Fellow, Institute for Policy Research,
The U.S. Supreme Court opined: “it is far worse to convict an innocent man than to let a guilty man go free”. How often are innocents convicted and guilty acquitted? These questions are difficult because the defendant’s “true state” rarely is known. Yet, as survey researchers know, even when truth cannot be known, a bound on inaccuracy can be estimated from well-designed data incorporating replication. E.g., sampling variance estimated from a simple random sample bounds mean squared error from below. In the context of criminal trials by jury, studies have obtained verdicts from not only the jury but also from the judge, shortly before the jury’s verdict announcement. Latent class models (LCMs) can be fitted to cross-tabulations of the data showing patterns of agreement and disagreement for different kinds of cases. In  I estimated type 1 and type 2 error rates, false conviction rates, false acquittal rates, and overall error rates from 2000-1 data developed by the National Center for State Courts.
I will review that analysis and adopt a total survey error (TSE) perspective to discuss statistical and non-statistical sources of error in LCM estimates of error rates. Measurement error arises, e.g., when the verdict reported by the judge differs from the ruling the judge would have reported in a bench trial. Identification error arises because agreement does not distinguish two correct from two incorrect verdicts. Specification error arises from omitted variables, incorrect LCM functional form, failure of independence assumptions, and measurement error in covariates such as “strength of evidence”. Even for correctly specified LCMs, type 1 and type 2 error rates are underestimated when the fitted latent class diverges from the desired concept of defendant’s “true state”. 
If society can only estimate error rates in verdicts with appreciable and imperfectly quantified uncertainty, should it do so or should it not even try? The question is addressed from the TSE perspective.