Generation of an annotated reference standard for vaccine adverse event reports

Matthew Foster, Abhishek Pandey, Kory Kreimeyer, Taxiarchis Botsis

Research output: Contribution to journalArticlepeer-review


As part of a collaborative project between the US Food and Drug Administration (FDA) and the Centers for Disease Control and Prevention for the development of a web-based natural language processing (NLP) workbench, we created a corpus of 1000 Vaccine Adverse Event Reporting System (VAERS) reports annotated for 36,726 clinical features, 13,365 temporal features, and 22,395 clinical-temporal links. This paper describes the final corpus, as well as the methodology used to create it, so that clinical NLP researchers outside FDA can evaluate the utility of the corpus to aid their own work. The creation of this standard went through four phases: pre-training, pre-production, production-clinical feature annotation, and production-temporal annotation. The pre-production phase used a double annotation followed by adjudication strategy to refine and finalize the annotation model while the production phases followed a single annotation strategy to maximize the number of reports in the corpus. An analysis of 30 reports randomly selected as part of a quality control assessment yielded accuracies of 0.97, 0.96, and 0.83 for clinical features, temporal features, and clinical-temporal associations, respectively and speaks to the quality of the corpus.

Original languageEnglish (US)
Pages (from-to)4325-4330
Number of pages6
Issue number29
StatePublished - Jul 5 2018


  • Annotation
  • Corpus
  • NLP
  • Reference

ASJC Scopus subject areas

  • Molecular Medicine
  • General Immunology and Microbiology
  • General Veterinary
  • Public Health, Environmental and Occupational Health
  • Infectious Diseases


Dive into the research topics of 'Generation of an annotated reference standard for vaccine adverse event reports'. Together they form a unique fingerprint.

Cite this