Program Session

5 - Replicating a Dr. Google Study: Challenges of Reproducing Online Information-Seeking Studies across Domains

Sunday, May 28
4:05 PM - 4:20 PM
Room: 607

Objectives: Reproducibility remains infrequently pursued for health information-seeking studies. Googling for a Diagnosis (BMJ, 2006) using New England Journal of Medicine cases has 350+ citations and inspired similar studies by medical specialties. We describe the challenges faced by a team comprising a specialist, a generalist, a student, and librarian in replicating the study in veterinary medicine while improving its methodological transparency.

Methods: In addition to addressing methodological concerns and limitations, we decided to explore the possible impact of clinician expertise on the searches and the final diagnoses. Starting in October 2014, we reviewed supplemental material for the detailed searches and discussed searching and expertise with several clinicians before deciding to have both a specialist and a generalist generate searches and diagnoses. The librarian and student selecting and blinding Journal of the American Veterinary Medical Association small animal medicine cases were independent from the clinicians developing Google searches and identifying diagnoses. We finished modifying the design in April 2015 and the changes included documenting clinicians’ previous exposure to the published cases, experience levels, and their individual search strategies. The student conducted all searches within a tight time window and the results were provided to both clinicians to inform their final diagnoses.

Results: The primary outcome of both studies was the percentage of Google-assisted correct diagnoses.  To address the limitations and questions raised in and about the BMJ paper findings, we compiled results on other factors to clarify the potential effect of the clinician on the search and therefore on the Google-assisted diagnostic performance.  We calculated pre-Googling diagnostic performance and confidence, documented the final diagnoses, and whether the diagnosis changed after the search process. We characterized search strategies by word count, term count, and inclusion of signalment, and described types of resources that appeared in the results. 

Conclusion:  Replicating the BMJ Google Diagnosis study was challenging due to the lack of specificity in the original article. Changing the domain increased potential confounders. Thirty small animal internal medicine cases came from five years of JAVMA compared to 26 from a single year of NEJM.  Veterinary information online remains scarce compared with human medicine. Finally, the nine year gap between studies raises questions about the impact that the growth of medical information and refinements of Google search algorithms likely have on comparison studies in terms of differences across domains and/or time periods.

Keywords: Reproducibility, health-information seeking, Google searching, case reports, diagnosis, veterinary medicine, expertise

Program Session

Kristine M. Alpi

Director, Wm. R. Kenan, Jr. Library of Veterinary Medicine
North Carolina State Univ. Libraries
North Carolina State University
Raleigh, North Carolina

Kristine M. Alpi, AHIP, is director of the William Rand Kenan, Jr. Library of Veterinary Medicine and adjunct assistant professor of population health and pathobiology at North Carolina State University. She is past-chair of the Research Section of MLA and has an active research portfolio. Her doctoral research on prior experience in health professions education will be completed in 2018.


Send Email for Kristine Alpi

George Schaaf

DVM Candidate
College of Veterinary Medicine
Raleigh, North Carolina


Send Email for George Schaaf


5 - Replicating a Dr. Google Study: Challenges of Reproducing Online Information-Seeking Studies across Domains

MP3 Audio Slides Video

Attendees who have favorited this

Please enter your access key

The asset you are trying to access is locked. Please enter your access key to unlock.

Send Email for Replicating a Dr. Google Study: Challenges of Reproducing Online Information-Seeking Studies across Domains