Scientific Results: Development and evaluation of a survey instrument to assess veterinary medical record suitability for multi-center research studies

August 23, 2022 - 7 minutes read

Posts in our Scientific Results series introduce recent papers published in the scientific literature by members of the Dog Aging Project research team. Follow this series to learn more about the scientific questions we’re asking, the kinds of results we’re getting, and what it all means for you and your dog.

Who worked on this research?

Dora Praczko*
Amanda Tinkle
Cryss Arkenberg
Robyn McClelland
Kate Creevy
Katherine Tolbert
Gray Barnett
Lucy Chou
Jeremy Evans
Kellyn McNulty
Dog Aging Project Consortium
Jonathan Levin

  • Praczko, Dora
  • amanda tinkle
  • Cryss Arkenberg, MWS, RLAT
  • Robyn
  • 07_Kate E. Creevy
  • 36_Katherine Tolbert
  • Gray Barnett bio photo - web
  • LucyChou
  • Evans
  • McNulty, Kellyn
  • cropped-DAP-favicon.png
  • 18_Jonathan Levine

Where was it published?

Frontiers in Veterinary Science, Comparative and Clinical Medicine

What is this paper about?

In this paper, we describe how the Dog Aging Project developed, assessed, and refined our veterinary electronic medical records (VEMR) evaluation process. Two sequential studies were completed to determine what screening criteria could be used to determine if an uploaded VEMR was both a true medical record and was suitable for research analysis.

Records were reviewed and scored during these studies by both board-certified internists and multiple paraprofessionals. The results were compared to ensure the Dog Aging Project was developing and utilizing a consistent, accurate veterinary records reviewing system. Veterinary medical records determined to be suitable for research are used to invite some of participants into smaller, nested studies (or cohorts) within the study, including our clinical trial.

The wide variety in veterinary medical records and the variation of those reviewing the records provided valuable insight into this process and led to a number of refinements. As we continued to improve the process, we developed extensive training, more descriptive questions, and a more streamlined process.

What do these results mean for me and my dog?

This paper provides a window into the inner workings of the Dog Aging Project, illustrating how much the Dog Aging Project puts into everything we do, including surveys that the participants do not even see. Here we see two pilot studies run in preparation for reviewing veterinary medical records. These pilots provided the Dog Aging Project with a new method for reviewing all of the veterinary medical records submitted to the Project.

Each step of the way, the project works to replicate this scientific rigor. Each survey and activity is tested to ensure it can be used in the best way possible to extend our knowledge of dogs. Many of our participant surveys have been in development and testing for years before they are seen by a participant. Like many things in life, it sometimes takes the fresh perspective of our participants to see something that improves the survey experience.

This process of reviewing and evaluating records is of great importance within the project and to our participants as it is one of the first steps in the screening process for consideration for a number of our smaller, nested studies. This evaluation process also provides a basis on which all records submitted to the Dog Aging Project were evaluated, giving us a consistent metric which can be utilized within the project itself, but also by the wider scientific community. Our hope is that other scientists can use this information in their own work moving forward and that it can prove beneficial when attempting to standardize veterinary records review for multiple scientific uses.

Where can I learn more?

Praczko D, Tinkle AK, Arkenberg CR, McClelland RL, Creevy KE, Tolbert MK, Barnett BG, Chou L, Evans J, Dog Aging Project Consortium, Levine JM. 2022. Development and evaluation of a survey instrument to assess veterinary medical record suitability for multi-center research studies Front. Vet. Sci. 9:941036. doi: 10.3389/fvets.2022.941036

Read the paper at Frontiers in Veterinary Science

Abstract

Here we describe the development and evaluation of a survey instrument to assess the research suitability of veterinary electronic medical records (EMRs) through the conduct of two studies as part of the Dog Aging Project (DAP). In study 1, four reviewers used the instrument to score a total of 218 records in an overlapping matrix of pairs to assess inter-rater agreement with respect to appropriate format (qualification), identification match (verification), and record quality. Based upon the moderate inter-rater agreement with respect to verification and the relatively large number of records that were incorrectly rejected the instrument was modified and more specific instructions were provided. In study 2, a modified instrument was again completed by four reviewers to score 100 different EMRs. The survey scores were compared to a gold standard of board-certified specialist review to determine receiver operating curve statistics. The refined survey had substantial inter-rater agreement across most qualification and verification questions. The cut-off value identified had a sensitivity of 95 and 96% (by reviewer 1 and reviewer 2, respectively) and a specificity of 82% and 91% (by reviewer 1 and reviewer 2, respectively) to predict gold standard acceptance or rejection of the record. Using just qualification and verification questions within the instrument (as opposed to full scoring) minimally impacted sensitivity and specificity and resulted in substantial time savings in the review process.

Tags: