Measuring Safety, Not Benchmark Scores: Infrastructural Barriers to Mental-Health AI Evaluation

Overview

This project began with interviewing 20 experts in the field as part of a National Science Foundation (NSF) program. As I progressed through the interviews, the data revealed clear recurring barriers to evaluation in the mental-health AI space. In this paper, we examine our findings and deliver clear recommendations to improving AI evaluation infrastructure for the mental health domain. Ongoing work involves getting participant approval to include anonymized quotes.

Abstract

Large language models are increasingly used for mental-health support, yet the field still lacks reliable methods for determining whether these systems are safe for vulnerable users. This paper examines the barriers preventing mental-health AI evaluation from producing evidence that is transparent and useful for governance. We draw on interviews with 20 benchmark developers, clinicians, lived-experience advocates, and policy researchers to identify recurring challenges in the design, maintenance, and use of mental-health safety evaluations. Across interviews, we find that progress is constrained by misaligned incentives, siloed evaluation tools, limited access to real-world interaction data, and uncertainty about how existing benchmark results should inform regulation. We also find that mental health is inherently pluralistic: definitions of harm vary across populations and clinical perspectives, making a universal safety standard both unrealistic and undesirable. We argue that the field should prioritize building a shared evaluation infrastructure with transparent comparative frameworks and sustained interdisciplinary collaboration so that different definitions of safety can be meaningfully compared rather than reduced to isolated benchmark scores.

On track for submission to FAccT 2027.