A further note on alternatives to Bpref (情報学基礎)
スポンサーリンク
概要
- 論文の詳細を見る
This paper compares the robustness of information retrieval (IR) metrics to incomplete relevance assessments, using four different sets of graded-relevance test collections with submitted runs-two from TREC and two from NTCIR. We investigate the effect of reducing the original relevance data on discriminative power (i.e., how often statistical significance can be detected given the probability of Type I Error) and on Kendall's rank correlation between two system rankings. According to these experiments, Q', nDCG' and AP' proposed by Sakai are superior to bpref proposed by Buckley and Voorhees and to Rank-Biased Precision proposed by Moffat and Zobel. We also clarify some properties of these metrics that immediately follow from their definitions.
- 一般社団法人情報処理学会の論文
- 2007-11-08
著者
-
SAKAI TETSUYA
Microsoft Research Asia
-
KANDO NORIKO
National Institute of Informatics
-
Kando Noriko
National Center For Science Information Systems
-
Sakai Tetsuya
NewsWatch, Inc.
-
Sakai Tetsuya
Newswatch Inc.
関連論文
- Ranking the NTCIR ACLIA IR4QA Systems without Relevance Assessments
- Revisiting NTCIR ACLIA IR4QA with Additional Relevance Assessments
- Revisiting NTCIR ACLIA IR4QA with Additional Relevance Assessments
- Ranking the NTCIR ACLIA IR4QA systems without relevance assessments
- Evaluating Information Retrieval Metrics Based on Bootstrap Hypothesis Tests
- Evaluation Methods for Web Retrieval Tasks Considering Hyperlink Structure(Special Issue on Text Processing for Information Access)
- Community QA Question Classification: Is the Asker Looking for Subjective Answers or Not?
- A further note on alternatives to Bpref (情報学基礎)
- A Graph-based Method for Automatic Generation of Multilingual Keyword Clusters and Its Applications
- Construction of Context Models for Word Sense Disambiguation
- Construction of context models for Word Sense Disambiguation ([SemEval-2日本語タスクを中心とする日本語語義曖昧性解消])
- On the Task of Finding One Highly Relevant Document with High Precision
- Comparing Metrics across TREC and NTCIR : The Robustness to System Bias
- Query Snowball: A Co-occurrence-based Approach to Multi-document Summarization for Question Answering
- Query Snowball: A Co-occurrence-based Approach to Multi-document Summarization for Question Answering
- Comparing metrics across TREC and NTCIR: the robustness to system bias (データベースシステム・情報学基礎)
- Web Search Evaluation with Informational and Navigational Intents
- Web Search Evaluation with Informational and Navigational Intents