A Further Note on Evaluation Metrics for the Task of Finding One Highly Relevant Document(情報検索・分類,テーマ : 「デジタルアーカイブの活用(応用)」および一般)
スポンサーリンク
概要
- 論文の詳細を見る
This paper proposes a new evaluation metric for Information Retrieval systems that aim at providing exactly one highly relevant document to the user. Such information retrieval tasks are especially important for modern large-scale retrieval environments (e.g. the Web) where recall is either unimportant or unknown. Existing metrics for the task of finding one relevant document assume that the user stops examining the ranked list of documents as soon as he finds one relevant document, even if it is a partially relevant one. In contrast, our proposed metric, called P-measure, assumes that the user looks for a highly relevant document even if it is ranked below partially relevant documents, and is probably suitable for retrieval situations such as known-item search or where it is easy for the user to spot a highly relevant document in the ranked output. Our main new findings, based on experiments using two sets of data comprising test collections and submitted runs from NTCIR, are: (a) P-measure is more stable and sensitive than Normalised Weighted Reciprocal Rank (NWRR) and Reciprocal Rank, and is at least as stable and sensitive as O-measure; and (b) Although O-measure and NWRR are highly correlated with each other, O-measure may be more stable and sensitive than NWRR. In summary, P-measure and O-measure are probably the most reliable metrics for the task of finding one highly relevant document. Researchers can decide on which one to use by considering which better models user behaviour in the real retrieval environment.
- 一般社団法人情報処理学会の論文
- 2006-03-22
著者
関連論文
- High-Precision Search via Question Abstraction for Japanese Question Answering
- High-Precision Search via Question Abstraction for Japanese Question Answering
- A Note on the Reliability of Japanese Question Answering Evaluation
- A Further Note on Evaluation Metrics for the Task of Finding One Highly Relevant Document(情報検索・分類,テーマ : 「デジタルアーカイブの活用(応用)」および一般)
- A Further Note on Evaluation Metrics for the Task of Finding One Highly Relevant Document(情報検索・分類,テーマ : 「デジタルアーカイブの活用(応用)」および一般)
- Controlling the Penalty on Late Arrival of Relevant Documents in Information Retrieval Evaluation with Graded Relevance