acl2016.pdf (197.99 kB)
A critique of word similarity as a method for evaluating distributional semantic models
conference contribution
posted on 2023-06-09, 02:11 authored by Miroslav Batchkarov, Thomas Kober, Jeremy ReffinJeremy Reffin, Julie WeedsJulie Weeds, David WeirDavid WeirThis paper aims to re-think the role of the word similarity task in distributional semantics research. We argue while it is a valuable tool, it should be used with care because it provides only an approximate measure of the quality of a distributional model. Word similarity evaluations assume there exists a single notion of similarity that is independent of a particular application. Further, the small size and low inter-annotator agreement of existing data sets makes it challenging to find significant differences between models.
Funding
A Unified Model of Compositional and Distributional Semantics: Theory and Applications; G0853; EPSRC-ENGINEERING & PHYSICAL SCIENCES RESEARCH COUNCIL; EP/IO37458/1
History
Publication status
- Published
File Version
- Accepted version
Journal
Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLPPublisher
Association for Computational LinguisticsExternal DOI
Page range
7-12Event name
The 1st Workshop on Evaluating Vector Space Representations for NLPEvent location
BerlinEvent type
conferenceEvent date
12th August 2016Place of publication
Berlin, GermanyDepartment affiliated with
- Informatics Publications
Full text available
- Yes
Peer reviewed?
- Yes