NTCIR Dialogue Evaluation Task




Last updated: February 25, 2021
Twitter: @ntcirdialeval

Please visit the webpage of our latest task: NTCIR-16 DialEval-2


Task definition slides updated May 17, 2020.
DialEval-1 (and WWW-3) task definitions in Japanese presented on September 10, 2019.
The task is a continuation of the Dialogue Quality (DQ) and Nugget Detection (ND) subtasks run at the NTCIR-14 STC-3 task (See References).

Important Dates (Timezone: Japan (UTC+9))

Jul 2019 Test data crawling [DONE]
Aug-Oct 2019 Adding more English translations to the training data [DONE]
Oct-Dec 2019 Test data annotation [DONE]
Jun 30, 2020 Test data released / Task registrations due [DONE]
Jul 31, 2020 Run submissions due [DONE]
Aug 31, 2020 Evaluation results, gold data, and draft overview paper released [DONE]
Sep 20, 2020 Draft participants papers due [DONE]
Oct 1, 2020 Task organisers’ feedback to participants [DONE]
Nov 1, 2020 All camera ready papers due [DONE]
Dec 8-11 2020 NTCIR-15 Conference [DONE]

Registration (NOW CLOSED)


To register, please send an email to dialeval1org@list.waseda.jp
with the following information so that we can send you the training data.
- Team Name (e.g. Waseda)
- Principal investigator’s name, affilication, email address
- Names, affiliations, email addresses of other team members
- Subtasks that you plan to participate: Chinese, English, or BOTH

Baseline run

Instead of building a system completely from scratch, participants may choose to utilise our baseline run. Details can be found HERE (TO BE ANNOUNCED).

Leaderboard with the STC-3 test data

Prior to run submission, participating teams can process the NTCIR-14 STC-3 test dialogues (NOT the new NTCIR-15 DialEval-1 test dialogues) and submit their runs to our online leaderboard, to see how well they perform. Note that annotations for the STC-3 test dialogues are given to registered participants, along with those for the STC-3 training data.

Details on how to submit to our leaderboard can be found HERE (TO BE ANNOUNCED).

Run submissions

Information on the run file names, run file format, the number of runs allowed per team, and where to submit can be found HERE (TO BE ANNOUNCED).

Organisers


Zhaohao Zeng (Waseda University, Japan)
Sosuke Kato (Waseda University, Japan)
Tetsuya Sakai (Waseda University, Japan)
Inho Kang (Naver Corporation, Korea)

INQUIRIES: dialeval1org@list.waseda.jp

References

Links


NTCIR-14 STC-3 wepage
NTCIR-15 webpage