Last updated: 22nd December, 2022.
NOTE: For our recent papers (2020-), links to the accompanying data can be found in our publication page or within the paper pdf.
Discpower for computing discriminative power using the randomised Tukey HSD test (Tetsuya Sakai)
NTCIREVAL for computing various IR effectiveness measures (Tetsuya Sakai)
NTCIRPOOL for making pool files for relevance assessments (Tetsuya Sakai)
BOOTS for conducting paired bootstrap tests (Tetsuya Sakai)
Data from Sakai’s SIGIR 2019 Paper (Which Diversity Evaluation Measures Are “Good”?)
Data from Sakai’s CLEF20 book chapter: How to Run an Evaluation Task
Data from Sakai’s SIGIR 2018 Short Paper (Comparing Two Binned Probability Distributions for Information Access Evaluation)
Tutorial kit for Sakai SIGIR 2018: Conducting Laboratory Experiments Properly with Statistical Tools: An Easy Hands-on Tutorial (includes the five excel files listed below besides other materials)
samplesizeANOVA2.xlsx for computing topic set sizes to achieve high statistical power for one-way ANOVA (recommended: See Sakai’s book)
samplesize2SAMPLECI.xlsx for computing topic set sizes to achieve a tight confidence interval for paired data (recommended: See Sakai’s book)
samplesize2SAMPLET.xlsx for computing topic set sizes to achieve high statistical power for the two-sample t-test
samplesizeTTEST2.xlsx for computing topic set sizes to achieve high statistical power for the paired t-test
<a href=”https://waseda.box.com/samplesizeCI2 for computing topic set sizes to achieve a tight confidence interval for unpaired data
Data from Sakai ICTIR 2016: A Simple and Effective Approach to Score Standardisation
Data and R code from Sakai SIGIR2016: Statistical Significance, Power, and Sample Sizes: A Systematic Review of SIGIR and TOIS, 2006-2015
Data from Sakai CIKM 2014: Designing Test Collections for Comparing Many Systems