Non-expert correction of automatically generated relation annotations

Matthew R. Gormley; Adam Gerber; Mary Harper; Mark Dredze

Non-expert correction of automatically generated relation annotations

Matthew R. Gormley, Adam Gerber, Mary Harper, Mark Dredze

Research output: Contribution to conference › Paper › peer-review

Abstract

We explore a new way to collect human annotated relations in text using Amazon Mechanical Turk. Given a knowledge base of relations and a corpus, we identify sentences which mention both an entity and an attribute that have some relation in the knowledge base. Each noisy sentence/relation pair is presented to multiple turkers, who are asked whether the sentence expresses the relation. We describe a design which encourages user efficiency and aids discovery of cheating. We also present results on inter-annotator agreement.

Original language	English (US)
Pages	204-207
Number of pages	4
State	Published - 2010
Externally published	Yes
Event	2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010 - Los Angeles, United States Duration: Jun 6 2010 → …

Conference

Conference	2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010
Country/Territory	United States
City	Los Angeles
Period	6/6/10 → …

ASJC Scopus subject areas

Language and Linguistics
Linguistics and Language

Cite this

Gormley, M. R., Gerber, A., Harper, M., & Dredze, M. (2010). Non-expert correction of automatically generated relation annotations. 204-207. Paper presented at 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010, Los Angeles, United States.

Non-expert correction of automatically generated relation annotations. / Gormley, Matthew R.; Gerber, Adam; Harper, Mary et al.
2010. 204-207 Paper presented at 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010, Los Angeles, United States.

Research output: Contribution to conference › Paper › peer-review

Gormley, MR, Gerber, A, Harper, M & Dredze, M 2010, 'Non-expert correction of automatically generated relation annotations', Paper presented at 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010, Los Angeles, United States, 6/6/10 pp. 204-207.

Gormley MR, Gerber A, Harper M, Dredze M. Non-expert correction of automatically generated relation annotations. 2010. Paper presented at 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010, Los Angeles, United States.

Gormley, Matthew R. ; Gerber, Adam ; Harper, Mary et al. / Non-expert correction of automatically generated relation annotations. Paper presented at 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010, Los Angeles, United States.4 p.

@conference{f50f87079e1a4127a98eca6f2847a549,

title = "Non-expert correction of automatically generated relation annotations",

abstract = "We explore a new way to collect human annotated relations in text using Amazon Mechanical Turk. Given a knowledge base of relations and a corpus, we identify sentences which mention both an entity and an attribute that have some relation in the knowledge base. Each noisy sentence/relation pair is presented to multiple turkers, who are asked whether the sentence expresses the relation. We describe a design which encourages user efficiency and aids discovery of cheating. We also present results on inter-annotator agreement.",

author = "Gormley, {Matthew R.} and Adam Gerber and Mary Harper and Mark Dredze",

note = "Publisher Copyright: {\textcopyright} 2010 Association for Computational Linguistics; 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010 ; Conference date: 06-06-2010",

year = "2010",

language = "English (US)",

pages = "204--207",

}

TY - CONF

T1 - Non-expert correction of automatically generated relation annotations

AU - Gormley, Matthew R.

AU - Gerber, Adam

AU - Harper, Mary

AU - Dredze, Mark

PY - 2010

Y1 - 2010

N2 - We explore a new way to collect human annotated relations in text using Amazon Mechanical Turk. Given a knowledge base of relations and a corpus, we identify sentences which mention both an entity and an attribute that have some relation in the knowledge base. Each noisy sentence/relation pair is presented to multiple turkers, who are asked whether the sentence expresses the relation. We describe a design which encourages user efficiency and aids discovery of cheating. We also present results on inter-annotator agreement.

AB - We explore a new way to collect human annotated relations in text using Amazon Mechanical Turk. Given a knowledge base of relations and a corpus, we identify sentences which mention both an entity and an attribute that have some relation in the knowledge base. Each noisy sentence/relation pair is presented to multiple turkers, who are asked whether the sentence expresses the relation. We describe a design which encourages user efficiency and aids discovery of cheating. We also present results on inter-annotator agreement.

UR - http://www.scopus.com/inward/record.url?scp=84878184911&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84878184911&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:84878184911

SP - 204

EP - 207

T2 - 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010

Y2 - 6 June 2010

ER -

Non-expert correction of automatically generated relation annotations

Abstract

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this