| アイテムタイプ |
学術雑誌論文 / Journal Article(1) |
| 公開日 |
2025-12-05 |
| タイトル |
|
|
タイトル |
Comparing Likert Scale and Pairwise Comparison for Human Evaluation in Rapport-Building Dialogue Systems |
| 言語 |
|
|
言語 |
eng |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
Human evaluation |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
Likert scale |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
pairwise comparison |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
rapport-building dialogue systems |
| 資源タイプ |
|
|
資源タイプ |
technical report |
| アクセス権 |
|
|
アクセス権 |
open access |
| 著者 |
Baihaqi, Muhammad Yeza
García, Contreras, Angel
Kawano, Seiya
吉野, 幸一郎
|
| 抄録 |
|
|
内容記述タイプ |
Abstract |
|
内容記述 |
Human evaluation plays a critical role in dialogue systems research, especially in non-task-oriented systems such as rapport-building dialogue systems. Current evaluations often rely on Likert scales to assess user experience, but this method introduces challenges such as inconsistent scale perception, inefficiency, and central tendency bias. Moreover, it is difficult to compare the agent's performance across multiple criteria due to the problem of uneven scoring interpretations by participants on the Likert scale. On the other hand, pairwise comparison emphasizes direct item-to-item evaluation based on defined criteria, producing scores that more closely align with participants' preferences and minimizing biases. This paper compares an evaluation framework for rapport-building dialogue systems using pairwise comparison with a conventional Likert scale system. These approaches are tested through dialogue experiments involving six participants and four dialogue systems embedded in a conversational robot: CommA, CommI, CommO, and CommE, to measure human-agent rapport. Our experimental results indicated that the pairwise comparison method better represented systems' overall performance compared to the Likert scale. It also demonstrated lower variability, higher reliability, and a shorter completion time. |
| 書誌情報 |
ja : 研究報告音声言語情報処理（SLP）
巻 2024-SLP-154,
号 43,
p. 1-5,
発行日 2024-12-05
|
| 出版者 |
|
|
出版者 |
情報処理学会 |
| ISSN |
|
|
収録物識別子タイプ |
EISSN |
|
収録物識別子 |
2188-8663 |
| 出版者版URI |
|
|
関連タイプ |
isVersionOf |
|
|
識別子タイプ |
URI |
|
|
関連識別子 |
https://ipsj.ixsq.nii.ac.jp/records/241663 |
| 権利 |
|
|
権利情報 |
ⓒ2024 Information Processing Society of Japan. ここに掲載した著作物の利用に関する注意 本著作物の著作権は情報処理学会に帰属します。本著作物は著作権者である情報処理学会の許可のもとに掲載するものです。ご利用に当たっては「著作権法」ならびに「情報処理学会倫理綱領」に従うことをお願いいたします。 Notice for the use of this material The copyright of this material is retained by the Information Processing Society of Japan (IPSJ). This material is published on this web site with the agreement of the author (s) and the IPSJ. Please be complied with Copyright Law of Japan and the Code of Ethics of the IPSJ if any users wish to reproduce, make derivative work, distribute or make available to the public any part or whole thereof. All Rights Reserved, Copyright (C) Information Processing Society of Japan. Comments are welcome. Mail to address editj＠ipsj.or.jp, please. |
| 著者版フラグ |
|
|
出版タイプ |
AM |
| 助成情報 |
|
|
|
助成機関名 |
Japan Society for the Promotion of Science (JSPS) |
|
|
研究課題番号 |
23K24910 |
|
|
研究課題番号URI |
https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-23K24910/ |
|
|
研究課題名 |
言語で記述された常識と実世界の観察を統合するロボットのための知識推論システム |
| 助成情報 |
|
|
|
助成機関名 |
Japan Society for the Promotion of Science (JSPS) |
|
|
研究課題番号 |
23K19984 |
|
|
研究課題番号URI |
https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-23K19984/ |
|
|
研究課題名 |
Designing an Expressive Relational Robotic Memory System with Long-Term Capabilities |
fulltext (1.4 MB)
