Last updated: February 25, 2021
Twitter:
@ntcirdialeval
Please visit the webpage of our latest task: NTCIR-16 DialEval-2
Task definition slides updated May 17, 2020.
DialEval-1 (and WWW-3) task definitions in Japanese presented on September 10, 2019.
The task is a continuation of the Dialogue Quality (DQ) and Nugget Detection (ND) subtasks run at the NTCIR-14 STC-3 task (See References).
Important Dates (Timezone: Japan (UTC+9))
| Jul 2019 |
Test data crawling [DONE]
|
| Aug-Oct 2019 |
Adding more English translations to the training data [DONE]
|
| Oct-Dec 2019 |
Test data annotation [DONE]
|
| Jun 30, 2020 |
Test data released / Task registrations due [DONE]
|
| Jul 31, 2020 |
Run submissions due [DONE]
|
| Aug 31, 2020 |
Evaluation results, gold data, and draft overview paper released [DONE]
|
| Sep 20, 2020 |
Draft participants papers due [DONE]
|
| Oct 1, 2020 |
Task organisers’ feedback to participants [DONE]
|
| Nov 1, 2020 |
All camera ready papers due [DONE]
|
| Dec 8-11 2020 |
NTCIR-15 Conference [DONE]
|
Registration (NOW CLOSED)
To register, please send an email to dialeval1org@list.waseda.jp
with the following information so that we can send you the training data.
- Team Name (e.g. Waseda)
- Principal investigator’s name, affilication, email address
- Names, affiliations, email addresses of other team members
- Subtasks that you plan to participate: Chinese, English, or BOTH
Baseline run
Instead of building a system completely from scratch, participants may choose to utilise our baseline run. Details can be found HERE (TO BE ANNOUNCED).
Leaderboard with the STC-3 test data
Prior to run submission, participating teams can process the NTCIR-14 STC-3 test dialogues (NOT the new NTCIR-15 DialEval-1 test dialogues) and submit their runs to our online leaderboard, to see how well they perform. Note that annotations for the STC-3 test dialogues are given to registered participants, along with those for the STC-3 training data.
Details on how to submit to our leaderboard can be found HERE (TO BE ANNOUNCED).
Run submissions
Information on the run file names, run file format, the number of runs allowed per team, and where to submit can be found HERE (TO BE ANNOUNCED).
Organisers
Zhaohao Zeng (Waseda University, Japan)
Sosuke Kato (Waseda University, Japan)
Tetsuya Sakai (Waseda University, Japan)
Inho Kang (Naver Corporation, Korea)
INQUIRIES: dialeval1org@list.waseda.jp
References
- Zeng, Z., Kato, S., and Sakai, T.: Overview of the NTCIR-14 Short Text Conversation Task: Dialogue Quality and Nugget Detection Subtasks, Proceedings of NTCIR-14, pp.289-315, 2019. pdf
- Sakai, T.: Comparing Two Binned Probability Distributions for Information Access Evaluation, Proceedings of ACM SIGIR 2018, pp.1073-1076, 2018. preprint pdf
- Zeng, Z., Luo, C., Shang, L., Li, H., and Sakai, T.: Towards Automatic Evaluation of Customer-Helpdesk Dialogues, Journal of Information Processing, Volume 26, pp.768-778, 2018. pdf
Links
NTCIR-14 STC-3 wepage
NTCIR-15 webpage