Abstract

Background
As artificial intelligence (AI) chatbots become an increasingly common source of quick medical guidance, it is important to understand whether their responses meet users’ needs and support well-informed health decisions. Yet, existing evaluation frameworks rely primarily on expert-defined evaluation dimensions that have not been empirically validated with end users. It remains unclear whether these frameworks capture the criteria people actually use when judging a response to be useful, trustworthy, and satisfying, or which evaluation dimensions matter most to users in practice.
Objective
This study empirically examined how commonly used dimensions such as Accuracy and Comprehensiveness shape caregivers’ satisfaction with AI-generated answers to pediatric health questions. We further investigated what expectations and communication needs may be overlooked by current evaluation frameworks.
Methods
We conducted a mixed methods vignette survey with 191 caregivers recruited through Prolific. Participants evaluated GPT-4o responses to a set of clinician-approved pediatric health questions across 8 dimensions and rated overall satisfaction. We quantified the influence of each dimension on overall satisfaction using a Cumulative Link Mixed Model and performed an inductive thematic analysis of open-ended comments to identify gaps in established frameworks.
Results
A total of 191 caregivers evaluated 1146 chatbot responses. Initially, caregivers rated Accuracy and Credibility as the most important dimensions. However, Cumulative Link Mixed Model analysis identified Usefulness as the strongest driver of overall …

Date

2026

Authors

Run Huang, Joseph Cecil, Marjorie Freedman, Souti Chattopadhyay

Journal

Journal of Medical Internet Research

Volume

28

Pages

e88126

Publisher

JMIR Publications