Comparing Metrics across TREC and NTCIR : The Robustness to System Bias
スポンサーリンク
概要
- 論文の詳細を見る
Test collections are growing larger, and relevance data constructed through pooling are suspected of becoming more and more incomplete and biased. Several studies have used evaluation metrics specifically designed to handle this problem, but most of them have only examined the metrics under incomplete but unbiased conditions, using random samples of the original relevance data. This paper examines nine metrics in more realistic settings, by reducing the number of pooled systems. Even though previous work has shown that metrics based on a condensed list, obtained by removing all unjudged documents from the original ranked list, are effective for handling very incomplete but unbiased relevance data, we show that they are not necessarily superior to traditional metrics in the presence of system bias. Using data from both TREC and NTCIR, we first show that condensed-list metrics overestimate new systems while traditional metrics underestimate them, and that the overestimation tends to be larger than the underestimation. We then show that, when relevance data is heavily biased towards a single team or a few teams, the condensed-list versions of Average Precision (AP), Q-measure (Q) and normalised Discounted Cumulative Gain (nDCG), which we call AP', Q' and nDCG', are not necessarily superior to the original metrics in terms of discriminative power, i.e., the overall ability to detect pairwise statistical significance. Nevertheless, AP' and Q' are generally more discriminative than bpref and the condensed-list version of Rank-Biased Precision (RBP), which we call RBP'.
- 一般社団法人情報処理学会の論文
- 2008-06-12
著者
関連論文
- Evaluating Information Retrieval Metrics Based on Bootstrap Hypothesis Tests
- A further note on alternatives to Bpref (情報学基礎)
- On the Task of Finding One Highly Relevant Document with High Precision
- Comparing Metrics across TREC and NTCIR : The Robustness to System Bias
- Comparing metrics across TREC and NTCIR: the robustness to system bias (データベースシステム・情報学基礎)