The Sakai Laboratory, Department of Computer Science and Engineering, Waseda University


  • sakailab.com
  • sakailab.com/english/
  • Professor Tetsuya Sakai

  • Data and tools for download


    Last updated: 12th June, 2018


    Tools provided at the NTCIR website

    Discpower for computing discriminative power using the randomised Tukey HSD test (Tetsuya Sakai)
    NTCIREVAL for computing various IR effectiveness measures (Tetsuya Sakai)
    NTCIRPOOL for making pool files for relevance assessments (Tetsuya Sakai)
    BOOTS for conducting paired bootstrap tests (Tetsuya Sakai)


    2019

    Data from Sakai’s CLEF20 book chapter: How to Run an Evaluation Task

    2018

    Tutorial kit for Sakai SIGIR 2018: Conducting Laboratory Experiments Properly with Statistical Tools: An Easy Hands-on Tutorial (includes the five excel files listed below besides other materials)
    samplesizeANOVA2.xlsx for computing topic set sizes to achieve high statistical power for one-way ANOVA (recommended: See Sakai’s book)
    samplesize2SAMPLECI.xlsx for computing topic set sizes to achieve a tight confidence interval for paired data (recommended: See Sakai’s book)
    samplesize2SAMPLET.xlsx for computing topic set sizes to achieve high statistical power for the two-sample t-test
    samplesizeTTEST2.xlsx for computing topic set sizes to achieve high statistical power for the paired t-test
    samplesizeCI2.xlsx for computing topic set sizes to achieve a tight confidence interval for unpaired data

    2017

    Code for paired and unpaired Bayesian tests (Sakai SIGIR 2017: The Probability That Your Hypothesis Is Correct, Credible Intervals, and Effect Sizes for IR Evaluation)

    2016

    Data from Sakai ICTIR 2016: A Simple and Effective Approach to Score Standardisation
    Data and R code from Sakai SIGIR2016: Statistical Significance, Power, and Sample Sizes: A Systematic Review of SIGIR and TOIS, 2006-2015

    2015

    2014

    Data from Sakai CIKM 2014: Designing Test Collections for Comparing Many Systems
    samplesizeTTEST.xlsx for computing topic set sizes to achieve high statistical power for the paired t-test (Sakai CIKM 2014: Designing Test Collections for Comparing Many Systems): This is an old version; use the above latest version instead
    samplesizeANOVA.xlsx for computing topic set sizes to achieve high statistical power for one-way ANOVA (Sakai CIKM 2014: Designing Test Collections for Comparing Many Systems): This is an old version; use the above latest version instead
    samplesizeCI.xlsx for computing topic set sizes to achieve a tight confidence interval for paired data (Sakai FIT 2014: Designing Test Collections That Provide Tight Confidence Intervals): This is an old version; use the above latest version instead

    2013

    Data from Sakai/Dou SIGIR 2013: Summaries, Ranked Retrieval and Sessions: A Unified Framework for Information Access Evaluation