Revisiting the case for explicit syntactic information in language models

Ariya Rastrow; Sanjeev Khudanpur; Mark Dredze

Revisiting the case for explicit syntactic information in language models

Ariya Rastrow, Sanjeev Khudanpur, Mark Dredze

Research output: Contribution to conference › Paper › peer-review

Abstract

Statistical language models used in deployed systems for speech recognition, machine translation and other human language technologies are almost exclusively n-gram models. They are regarded as linguistically naïve, but estimating them from any amount of text, large or small, is straightforward. Furthermore, they have doggedly matched or outperformed numerous competing proposals for syntactically well-motivated models. This unusual resilience of n-grams, as well as their weaknesses, are examined here. It is demonstrated that n-grams are good word-predictors, even linguistically speaking, in a large majority of word-positions, and it is suggested that to improve over n-grams, one must explore syntax-aware (or other) language models that focus on positions where n-grams are weak.

Original language	English (US)
Pages	50-58
Number of pages	9
State	Published - 2012
Externally published	Yes
Event	2012 Workshop on Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, WLM 2012 at the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2012 - Montreal, Canada Duration: Jun 8 2012 → …

Conference

Conference	2012 Workshop on Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, WLM 2012 at the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2012
Country/Territory	Canada
City	Montreal
Period	6/8/12 → …

ASJC Scopus subject areas

Linguistics and Language
Language and Linguistics
Computer Science Applications

Cite this

Rastrow, A., Khudanpur, S., & Dredze, M. (2012). Revisiting the case for explicit syntactic information in language models. 50-58. Paper presented at 2012 Workshop on Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, WLM 2012 at the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2012, Montreal, Canada.

Revisiting the case for explicit syntactic information in language models. / Rastrow, Ariya; Khudanpur, Sanjeev; Dredze, Mark.
2012. 50-58 Paper presented at 2012 Workshop on Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, WLM 2012 at the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2012, Montreal, Canada.

Research output: Contribution to conference › Paper › peer-review

Rastrow, A, Khudanpur, S & Dredze, M 2012, 'Revisiting the case for explicit syntactic information in language models', Paper presented at 2012 Workshop on Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, WLM 2012 at the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2012, Montreal, Canada, 6/8/12 pp. 50-58.

Rastrow A, Khudanpur S, Dredze M. Revisiting the case for explicit syntactic information in language models. 2012. Paper presented at 2012 Workshop on Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, WLM 2012 at the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2012, Montreal, Canada.

Rastrow, Ariya ; Khudanpur, Sanjeev ; Dredze, Mark. / Revisiting the case for explicit syntactic information in language models. Paper presented at 2012 Workshop on Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, WLM 2012 at the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2012, Montreal, Canada.9 p.

@conference{b621c55380f4429c94c28ceeed71f921,

title = "Revisiting the case for explicit syntactic information in language models",

abstract = "Statistical language models used in deployed systems for speech recognition, machine translation and other human language technologies are almost exclusively n-gram models. They are regarded as linguistically na{\"i}ve, but estimating them from any amount of text, large or small, is straightforward. Furthermore, they have doggedly matched or outperformed numerous competing proposals for syntactically well-motivated models. This unusual resilience of n-grams, as well as their weaknesses, are examined here. It is demonstrated that n-grams are good word-predictors, even linguistically speaking, in a large majority of word-positions, and it is suggested that to improve over n-grams, one must explore syntax-aware (or other) language models that focus on positions where n-grams are weak.",

author = "Ariya Rastrow and Sanjeev Khudanpur and Mark Dredze",

note = "Publisher Copyright: {\textcopyright} 2012 Association for Computational Linguistics.; 2012 Workshop on Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, WLM 2012 at the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2012 ; Conference date: 08-06-2012",

year = "2012",

language = "English (US)",

pages = "50--58",

}

TY - CONF

T1 - Revisiting the case for explicit syntactic information in language models

AU - Rastrow, Ariya

AU - Khudanpur, Sanjeev

AU - Dredze, Mark

PY - 2012

Y1 - 2012

N2 - Statistical language models used in deployed systems for speech recognition, machine translation and other human language technologies are almost exclusively n-gram models. They are regarded as linguistically naïve, but estimating them from any amount of text, large or small, is straightforward. Furthermore, they have doggedly matched or outperformed numerous competing proposals for syntactically well-motivated models. This unusual resilience of n-grams, as well as their weaknesses, are examined here. It is demonstrated that n-grams are good word-predictors, even linguistically speaking, in a large majority of word-positions, and it is suggested that to improve over n-grams, one must explore syntax-aware (or other) language models that focus on positions where n-grams are weak.

AB - Statistical language models used in deployed systems for speech recognition, machine translation and other human language technologies are almost exclusively n-gram models. They are regarded as linguistically naïve, but estimating them from any amount of text, large or small, is straightforward. Furthermore, they have doggedly matched or outperformed numerous competing proposals for syntactically well-motivated models. This unusual resilience of n-grams, as well as their weaknesses, are examined here. It is demonstrated that n-grams are good word-predictors, even linguistically speaking, in a large majority of word-positions, and it is suggested that to improve over n-grams, one must explore syntax-aware (or other) language models that focus on positions where n-grams are weak.

UR - http://www.scopus.com/inward/record.url?scp=84905675561&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84905675561&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:84905675561

SP - 50

EP - 58

T2 - 2012 Workshop on Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, WLM 2012 at the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2012

Y2 - 8 June 2012

ER -

Revisiting the case for explicit syntactic information in language models

Abstract

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this