Last updated: October 10, 2019
Task definition slides
updated October 16, 2019.
DialEval-1 (and WWW-3) task definitions in Japanese
presented on September 10, 2019.
The task is a continuation of the Dialogue Quality (DQ) and Nugget Detection (ND) subtasks run at the NTCIR-14 STC-3 task (See References).
Important Dates (Timezone: Japan (UTC+9))
|| Test data crawling [DONE]
|| Adding more English translations to the training data [DONE]
|| Task registrations open
|| Test data annotation
|| Test data released / Task registrations due
|| Run submissions due
|| Evaluation results released
|| NTCIR-15 Conference at NII, Tokyo, Japan
Registration (IN PREPARATION)
To register, please send an email to firstname.lastname@example.org
with the following information so that we can send you the training data.
- Team Name (e.g. Waseda)
- Principal investigator’s name, affilication, email address
- Names, affiliations, email addresses of other team members
- Subtasks that you plan to participate: Chinese, English, or BOTH
(Later, NII will require you to register to NTCIR tasks through their website, but please contact us by email first.
Baseline run (IN PREPARATION)
Instead of building a system completely from scratch, participants may choose to utilise our baseline run. Details can be found HERE (TO BE ANNOUNCED).
Leaderboard with the STC-3 test data (IN PREPARATION)
Prior to run submission, participating teams can process the NTCIR-14 STC-3 test dialogues (NOT the new NTCIR-15 DialEval-1 test dialogues) and submit their runs to our online leaderboard, to see how well they perform. Note that annotations for the STC-3 test dialogues are given to registered participants, along with those for the STC-3 training data.
Details on how to submit to our leaderboard can be found HERE (TO BE ANNOUNCED).
Run submissions (IN PREPARATION)
Information on the run file names, run file format, the number of runs allowed per team, and where to submit can be found HERE (TO BE ANNOUNCED).
Zhaohao Zeng (Waseda University, Japan)
Sosuke Kato (Waseda University, Japan)
Tetsuya Sakai (Waseda University, Japan)
Inho Kang (Naver Corporation, Korea)
- Zeng, Z., Kato, S., and Sakai, T.: Overview of the NTCIR-14 Short Text Conversation Task: Dialogue Quality and Nugget Detection Subtasks, Proceedings of NTCIR-14, pp.289-315, 2019. pdf
- Sakai, T.: Comparing Two Binned Probability Distributions for Information Access Evaluation, Proceedings of ACM SIGIR 2018, pp.1073-1076, 2018. preprint pdf
- Zeng, Z., Luo, C., Shang, L., Li, H., and Sakai, T.: Towards Automatic Evaluation of Customer-Helpdesk Dialogues, Journal of Information Processing, Volume 26, pp.768-778, 2018. pdf
NTCIR-14 STC-3 wepage