Non-expert correction of automatically generated relation annotations

Matthew R. Gormley, Adam Gerber, Mary Harper, Mark Dredze

Research output: Contribution to conferencePaperpeer-review


We explore a new way to collect human annotated relations in text using Amazon Mechanical Turk. Given a knowledge base of relations and a corpus, we identify sentences which mention both an entity and an attribute that have some relation in the knowledge base. Each noisy sentence/relation pair is presented to multiple turkers, who are asked whether the sentence expresses the relation. We describe a design which encourages user efficiency and aids discovery of cheating. We also present results on inter-annotator agreement.


Conference2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010
Country/TerritoryUnited States
CityLos Angeles
Period6/6/10 → …

ASJC Scopus subject areas

  • Language and Linguistics
  • Linguistics and Language


Dive into the research topics of 'Non-expert correction of automatically generated relation annotations'. Together they form a unique fingerprint.

Cite this