A Closer Look at Claim Decomposition

Miriam Wanner, Seth Ebner, Zhengping Jiang, Mark Dredze, Benjamin Van Durme

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

As generated text becomes more commonplace, it is increasingly important to evaluate how well-supported such text is by external knowledge sources. Many approaches for evaluating textual support rely on some method for decomposing text into its individual subclaims which are scored against a trusted reference. We investigate how various methods of claim decomposition—especially LLM-based methods—affect the result of an evaluation approach such as the recently proposed FACTSCORE, finding that it is sensitive to the decomposition method used. This sensitivity arises because such metrics attribute overall textual support to the model that generated the text even though error can also come from the metric’s decomposition step. To measure decomposition quality, we introduce an adaptation of FACTSCORE, which we call DECOMPSCORE. We then propose an LLM-based approach to generating decompositions inspired by Bertrand Russell’s theory of logical atomism and neo-Davidsonian semantics and demonstrate its improved decomposition quality over previous methods.

Original languageEnglish (US)
Title of host publicationStarSEM 2024 - 13th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference
EditorsDanushka Bollegala, Danushka Bollegala, Vered Shwartz
PublisherAssociation for Computational Linguistics (ACL)
Pages153-175
Number of pages23
ISBN (Electronic)9798891761063
DOIs
StatePublished - 2024
Event13th Joint Conference on Lexical and Computational Semantics, StarSEM 2024 - Mexico City, Mexico
Duration: Jun 20 2024Jun 21 2024

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)0736-587X

Conference

Conference13th Joint Conference on Lexical and Computational Semantics, StarSEM 2024
Country/TerritoryMexico
CityMexico City
Period6/20/246/21/24

ASJC Scopus subject areas

  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Fingerprint

Dive into the research topics of 'A Closer Look at Claim Decomposition'. Together they form a unique fingerprint.

Cite this