NTCIR Dialogue Evaluation Task




Last updated: October 10, 2019
Twitter: @ntcirdialeval
Task definition slides updated October 16, 2019.
DialEval-1 (and WWW-3) task definitions in Japanese presented on September 10, 2019.
The task is a continuation of the Dialogue Quality (DQ) and Nugget Detection (ND) subtasks run at the NTCIR-14 STC-3 task (See References).

Important Dates (Timezone: Japan (UTC+9))

Jul 2019 Test data crawling [DONE]
Aug-Oct 2019 Adding more English translations to the training data [DONE]
Oct 2019 Task registrations open
Oct-Dec 2019 Test data annotation
Jun 2020 Test data released / Task registrations due
Jul 2020 Run submissions due
Aug 2020 Evaluation results released
Dec 2020 NTCIR-15 Conference at NII, Tokyo, Japan

Registration (IN PREPARATION)


To register, please send an email to dialeval1org@list.waseda.jp
with the following information so that we can send you the training data.
- Team Name (e.g. Waseda)
- Principal investigator’s name, affilication, email address
- Names, affiliations, email addresses of other team members
- Subtasks that you plan to participate: Chinese, English, or BOTH
(Later, NII will require you to register to NTCIR tasks through their website, but please contact us by email first.)

Baseline run (IN PREPARATION)

Instead of building a system completely from scratch, participants may choose to utilise our baseline run. Details can be found HERE (TO BE ANNOUNCED).

Leaderboard with the STC-3 test data (IN PREPARATION)

Prior to run submission, participating teams can process the NTCIR-14 STC-3 test dialogues (NOT the new NTCIR-15 DialEval-1 test dialogues) and submit their runs to our online leaderboard, to see how well they perform. Note that annotations for the STC-3 test dialogues are given to registered participants, along with those for the STC-3 training data.

Details on how to submit to our leaderboard can be found HERE (TO BE ANNOUNCED).

Run submissions (IN PREPARATION)

Information on the run file names, run file format, the number of runs allowed per team, and where to submit can be found HERE (TO BE ANNOUNCED).

Organisers


Zhaohao Zeng (Waseda University, Japan)
Sosuke Kato (Waseda University, Japan)
Tetsuya Sakai (Waseda University, Japan)
Inho Kang (Naver Corporation, Korea)

INQUIRIES: dialeval1org@list.waseda.jp

References

Links


NTCIR-14 STC-3 wepage
NTCIR-15 webpage