This paper aims to re-think the role of the word similarity task in distributional semantics research. We argue while it is a valuable tool, it should be used with care because it provides only an approximate measure of the quality of a distributional model. Word similarity evaluations assume there exists a single notion of similarity that is independent of a particular application. Further, the small size and low inter-annotator agreement of existing data sets makes it challenging to find significant differences between models.
Funding
A Unified Model of Compositional and Distributional Semantics: Theory and Applications; G0853; EPSRC-ENGINEERING & PHYSICAL SCIENCES RESEARCH COUNCIL; EP/IO37458/1
History
Publication status
Published
File Version
Accepted version
Journal
Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP