Evaluating GPT-4o's Performance and Errors on a Chilean Anesthesiology Board Exam

“

For Healthcare Leaders and Administrators: This study provides evidence-based guidance for AI implementation decisions. GPT-4o's performance suggests potential applications in clinical documentation support, medical education, and knowledge retrieval, but implementation requires substantial investment in verification systems, training protocols, and oversight mechanisms. The 16.3% error rate demands multi-layered safety protocols before clinical deployment. For Clinicians and Anesthesiologists: While AI demonstrates impressive knowledge recall, this research confirms that human clinical judgment remains irreplaceable. The prevalence of unsupported medical claims (41% of errors) means AI outputs require careful verification, particularly for complex diagnostic and treatment decisions. AI may serve best as a supplementary tool for literature review, initial documentation drafting, and educational support—always with physician oversight. For Medical Educators: The performance gap between basic recall (84%) and complex application (77%) suggests AI could supplement medical education for foundational learning while requiring enhanced teaching of higher-order clinical reasoning skills. These findings underscore the need to integrate AI literacy into medical curricula while emphasizing critical evaluation of AI-generated recommendations. For Policymakers and Regulators: This research demonstrates the necessity for specialty-specific AI validation, robust liability frameworks, and standardized safety protocols. The findings support implementing accuracy thresholds of 80% for educational applications and 90% for clinical decision support, with mandatory human oversight regardless of AI performance levels. International coordination on evaluation standards is essential. For AI Developers: The error taxonomy reveals specific technical challenges requiring attention: knowledge verification mechanisms to address unsupported claims, improved reasoning chains to prevent vague conclusions, and domain-specific fine-tuning to enhance performance on complex analytical tasks. The minimal impact of temperature parameter optimization suggests focusing development efforts on fundamental reasoning improvements rather than parameter tuning. For Researchers: This work establishes a methodological framework for rigorous LLM evaluation in medical specialties, including structured error taxonomies, multi-rater annotation protocols, and cross-linguistic assessment approaches. Priority research needs include cross-specialty transferability studies, long-term clinical outcome evaluations, and human-AI collaboration optimization research. For Patients and the Public: While AI shows promise for improving healthcare access and supporting medical decision-making, this research confirms that AI systems make meaningful errors that could affect patient safety. Current AI technology requires doctor oversight and should be viewed as a tool to assist—not replace—human medical expertise. Patients should expect transparency about AI use in their care and continued physician accountability for medical decisions.

”