Last updated: 21st March 2024.
Twitter: @ntcirfairweb

---

Updates

• NTCIR-17 FairWeb-1 official results (with qrels etc.) are here.
  NTCIR-17 FairWeb-1 is over. Please be ready for NTCIR-18 FairWeb-2!
• Video of FairWeb-1 overview talk by Sijie Tao YouTube (from around 27:00)
• Video of the FairWeb-1 task session (featuring talks by Tsinghua Uni, Waseda Uni, Uni of Delaware, and RMIT uni) YouTube (59 minutes)
• The RMIT team won the Best Presation Award of the FairWeb-1 task! Congrats!
• Sorry for the lack of clarity: the target distribution for the ORIGIN attribute set for M-topics at the bottom of this excel file (the green area; based on the number of countries/regions). For all other attribute sets, we will use uniform target distributions. (April 22)
• Test topics can be found here! Run submission instructions can be found here! Baseline runs with the html files (for ReRanked runs) can be found here! (March 15)
  (NOTE: We might drop some topics for the official evaluation if we fail to find relevant entities for them.)
• Target corpus announced! Our task will use Chuweb21D-60 (49.8M web pages). Available from the Chuweb21D page! (February 27, 2023)
• Pilot data can be found here! This package demonstrates how our pilot runs (for four sample topics) were actually evaluated using our framework. (February 15, 2023)
• Here are the four sample topics! (October 3, 2022)

---

Task Specifications

Introduction to FairWeb-1 (slides; updated Feb 2023)

---

Timeline (GMT+9 except where indicated)

blue: what participants will do
red: what organisers will do

October 3, 2022	Release of 1st CFP with sample topics and evaluation protocol [DONE]
February 15, 2023	Pilot relevance assessments for the sample topics and a few pilot runs released [DONE]
February 1-March 10, 2023	Topic development [DONE]
March 15, 2023	Topics released [DONE]
April 15, 2023	Task registrations due (Anywhere on Earth). Sign up from this NTCIR-17 page! [DONE]
May 22, 2023	Run submissions due [DONE]
May 23-July 31, 2023	Entity annotations; runs evaluated [DONE]
August 1, 2023	Evaluation results and draft overview released [DONE]
September 1, 2023	Draft participant papers due [DONE]
November 1, 2023	Camera ready papers due
December 12-15 2023	NTCIR-17@NII, Tokyo, Japan

---

Papers

Sijie Tao, Nuo Chen, Tetsuya Sakai, Zhumin Chu, Hiromi Arai, Ian Soboroff, Nicola Ferro, Maria Maistro: Overview of the NTCIR-17 FairWeb-1 Task, Proceedings of NTCIR-17, pp.284-305, 2023. pdf

Zhumin Chu, Tetsuya Sakai, Qingyao Ai, and Yiqun Liu: Chuweb21D: A Deduped English Document Collection for Web Search Tasks, Proceedings of ACM SIGIR-AP 2023, pp.63-72, 2023. pdf

Tetsuya Sakai, Jin Young Kim, and Inho Kang: A Versatile Framework for Evaluating Ranked Lists in terms of Group Fairness and Relevance, ACM TOIS, 2023. pdf

(In Japanese) 酒井哲也: グループフェアネスを考慮したウェブ検索タスク, 情報処理学会研究報告 2022-DBS-175/2022-IFAT-148, No.6, 2022. paper slides

---

Links

NTCIR
TREC Fair Ranking Track

---

Organisers

Email: fairweb1org at
list.waseda.jp

Sijie Tao, Nuo Chen, Tetsuya Sakai (Waseda University, Japan)
Zhumin Chu (Tsinghua University, P.R.C.)
Nicola Ferro (University of Padua, Italy)
Maria Maistro (University of Copenhagen, Denmark)
Ian Soboroff (NIST, USA)
Hiromi Arai (RIKEN AIP, Japan)

---