| アイテムタイプ |
学術雑誌論文 / Journal Article(1) |
| 公開日 |
2025-09-11 |
| タイトル |
|
|
タイトル |
Evaluating and Enhancing Japanese Large Language Models for Genetic Counseling Support: Comparative Study of Domain Adaptation and the Development of an Expert-Evaluated Dataset |
| 言語 |
|
|
言語 |
eng |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
large language models |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
genetic counseling |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
medical |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
health |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
artificial intelligence |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
machine learning |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
domain adaptation |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
retrieval-augmented generation |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
instruction tuning |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
prompt engineering |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
question-answer |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
dialogue |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
ethics |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
safety |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
low-rank adaptation |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
Japanese |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
expert evaluation |
| 資源タイプ |
|
|
資源タイプ |
journal article |
| アクセス権 |
|
|
アクセス権 |
open access |
| 著者 |
Fukushima, Takuya
Manabe, Masae
矢田, 竣太郎
若宮, 翔子
Yoshida, Akiko
Urakawa, Yusaku
Maeda, Akiko
Kan, Shigeyuki
Takahashi, Masayo
荒牧, 英治
|
| 抄録 |
|
|
内容記述タイプ |
Abstract |
|
内容記述 |
Background: Advances in genetics have underscored a strong association between genetic factors and health outcomes, leading to an increased demand for genetic counseling services. However, a shortage of qualified genetic counselors poses a significant challenge. Large language models (LLMs) have emerged as a potential solution for augmenting support in genetic counseling tasks. Despite the potential, Japanese genetic counseling LLMs (JGCLLMs) are underexplored. To advance a JGCLLM-based dialogue system for genetic counseling, effective domain adaptation methods require investigation. Objective: This study aims to evaluate the current capabilities and identify challenges in developing a JGCLLM-based dialogue system for genetic counseling. The primary focus is to assess the effectiveness of prompt engineering, retrieval-augmented generation (RAG), and instruction tuning within the context of genetic counseling. Furthermore, we will establish an experts-evaluated dataset of responses generated by LLMs adapted to Japanese genetic counseling for the future development of JGCLLMs. Methods: Two primary datasets were used in this study: (1) a question-answer (QA) dataset for LLM adaptation and (2) a genetic counseling question dataset for evaluation. The QA dataset included 899 QA pairs covering medical and genetic counseling topics, while the evaluation dataset contained 120 curated questions across 6 genetic counseling categories. Three enhancement techniques of LLMs---instruction tuning, RAG, and prompt engineering---were applied to a lightweight Japanese LLM to enhance its ability for genetic counseling. The performance of the adapted LLM was evaluated on the 120-question dataset by 2 certified genetic counselors and 1 ophthalmologist (SK, YU, and AY). Evaluation focused on four metrics: (1) inappropriateness of information, (2) sufficiency of information, (3) severity of harm, and (4) alignment with medical consensus. Results: The evaluation by certified genetic counselors and an ophthalmologist revealed varied outcomes across different methods. RAG showed potential, particularly in enhancing critical aspects of genetic counseling. In contrast, instruction tuning and prompt engineering produced less favorable outcomes. This evaluation process facilitated the creation an expert-evaluated dataset of responses generated by LLMs adapted with different combinations of these methods. Error analysis identified key ethical concerns, including inappropriate promotion of prenatal testing, criticism of relatives, and inaccurate probability statements. Conclusions: RAG demonstrated notable improvements across all evaluation metrics, suggesting potential for further enhancement through the expansion of RAG data. The expert-evaluated dataset developed in this study provides valuable insights for future optimization efforts. However, the ethical issues observed in JGCLLM responses underscore the critical need for ongoing refinement and thorough ethical evaluation before these systems can be implemented in health care settings. |
| 書誌情報 |
en : JMIR Medical Informatics
巻 13,
ページ数 16,
発行日 2025-01-16
|
| 出版者 |
|
|
出版者 |
JMIR Publications |
| ISSN |
|
|
収録物識別子タイプ |
EISSN |
|
収録物識別子 |
2291-9694 |
| 出版者版DOI |
|
|
関連タイプ |
isReplacedBy |
|
|
識別子タイプ |
DOI |
|
|
関連識別子 |
https://doi.org/10.2196/65047 |
| 出版者版URI |
|
|
関連タイプ |
isReplacedBy |
|
|
識別子タイプ |
URI |
|
|
関連識別子 |
https://medinform.jmir.org/2025/1/e65047 |
| 権利 |
|
|
権利情報Resource |
https://creativecommons.org/licenses/by/4.0/ |
|
権利情報 |
©Takuya Fukushima, Masae Manabe, Shuntaro Yada, Shoko Wakamiya, Akiko Yoshida, Yusaku Urakawa, Akiko Maeda, Shigeyuki Kan, Masayo Takahashi, Eiji Aramaki. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 16.01.2025. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included. |
| 著者版フラグ |
|
|
出版タイプ |
NA |
| 助成情報 |
|
|
|
助成機関名 |
Japan Science and Technology Agency (JST) |
|
|
研究課題番号 |
JPMJCR22N1 |
|
|
研究課題番号URI |
https://projectdb.jst.go.jp/grant/JST-PROJECT-22717060/ |
|
|
研究課題名 |
リアルワールドテキスト処理の深化によるデータ駆動型探薬 |
| 助成情報 |
|
|
|
助成機関名 |
National Center for Global Health and Medicine(NCGM) |
|
|
研究課題番号 |
JPJ012425 |
|
|
研究課題名 |
Integrated Health Care System |
