Workshop 9: Small Data and Big Data

The 9th 2013/2014 Workshop on Quantitative Methods in Education, Health and the Social Sciences (QMEHSS) will be on Friday, May 16th from 11:00-12:30 and will be led by Dr. Naihua Duan. The seminar will be held in the NORC conference room 344. NORC is located at 1155 E. 60th Street.


Small Data and Big Data

Naihua Duan, Ph.D.

Division of Biostatistics, Department of Psychiatry, Columbia University

The “big data” enterprise emerged in recent years as a powerful paradigm for clinical and policy decision-making, facilitated by the rapid growth in information technology.  It is employed widely in the private sector for targeted marketing, as most regular users of will likely testify.  In the public sector, the U.S. Federal Government launched the “Big Data Research and Development Initiative” in March 2012, with the aim to “transform our ability to use Big Data for scientific discovery, environmental and biomedical research, education, and national security.”  There is an extensive interest in the potential consequences in the deployment of the “big data” paradigm.

At the same time, the “small data” enterprise also emerged recently as a supplement, or alternative, to the “big data” enterprise.  While the “big data” enterprise is usually focused on serving the supplier or regulator, the “small data” enterprise is focused on serving end-users, such as consumers, in a direct way.  Pollock (2013) argued that we should “Forget big data, small data is the real revolution”, with the reasoning that the real revolution in information technology “is the mass democratisation of the means of access, storage and processing of data.”  Estrin (2013) observed: “We leave a trail of digital data crumbs as we go about our days.  With access and good apps, we could make sense of this “small data” to help get a clearer picture of our personal health.”  This desire among consumers to “get a clearer picture of our personal health” has fostered several “small data” movements, such as and, that facilitate patients to keep track and makes sense of their own “data crumbs”.

I will discuss the opportunities for quantitative methodologists in the “small data” enterprise, as a rich domain for innovative and potentially high impact developments of methodologies and applications, with illustrations from Cheung and Duan (2014), Duan, Kravitz and Schmid (2013), and Kravitz and Duan (2014).

Cheung K, Duan N. Design of implementation studies for quality improvement programs: an effectiveness-cost-effectiveness framework. Am J Public Health. 2014 Jan;104(1):e23-30.  Epub 2013 Nov 14.

Duan N, Kravitz RL, Schmid CH. Single-patient (n-of-1) trials: a pragmatic clinical decision methodology for patient-centered comparative effectiveness research.  J Clin Epidemiol. 2013 Aug;66(8 Suppl):S21-8.

Kravitz RL, Duan N, eds, and the DEcIDE Methods Center N-of-1 Guidance Panel (Duan N, Eslick I, Gabler NB, Kaplan HC, Kravitz RL, Larson EB, Pace WD, Schmid CH, Sim I, Vohra S). Design and Implementation of N-of-1 Trials: A User’s Guide. AHRQ Publication No. 13(14)-EHC122-EF. Rockville, MD: Agency for Healthcare Research and Quality; February 2014.