Exploiting feature covariance in high-dimensional online learning

Justin Ma, Alex Kulesza, Mark Dredze, Koby Crammer, Lawrence K. Saul, Fernando Pereira

Research output: Contribution to journalConference articlepeer-review

Abstract

Some online algorithms for linear classification model the uncertainty in their weights over the course of learning. Modeling the full covariance structure of the weights can provide a significant advantage for classification. However, for high-dimensional, largescale data, even though there may be many second-order feature interactions, it is computationally infeasible to maintain this covariance structure. To extend second-order methods to high-dimensional data, we develop low-rank approximations of the covariance structure. We evaluate our approach on both synthetic and real-world data sets using the confidence-weighted (Dredze et al., 2008; Crammer et al., 2009a) online learning framework. We show improvements over diagonal covariance matrices for both low and high-dimensional data.

Original languageEnglish (US)
Pages (from-to)493-500
Number of pages8
JournalJournal of Machine Learning Research
Volume9
StatePublished - 2010
Event13th International Conference on Artificial Intelligence and Statistics, AISTATS 2010 - Sardinia, Italy
Duration: May 13 2010May 15 2010

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Statistics and Probability
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'Exploiting feature covariance in high-dimensional online learning'. Together they form a unique fingerprint.

Cite this