What is it about?
AI systems like GPT-4o and Gemini can now look at a photo and answer questions about it. They are usually judged on one thing: whether they get the answer right. But when these systems describe real people in news photos, being "right" isn't enough. They also need to be fair across groups, avoid harmful stereotypes, respond with care in sensitive situations, work well in languages other than English, and keep working when images are blurry or noisy. We built HumaniBench, a test suite of about 32,000 questions about real news images, checked by human experts. It measures AI models on seven human-centred qualities: fairness, ethics, understanding, reasoning, language inclusivity, empathy, and robustness. We tested 15 leading models. Commercial models like GPT-4o did best on ethics, reasoning, and empathy. Some free, open-source models were better at pinpointing objects in images and handling poor-quality photos. No model did well on everything. All models were less accurate on questions involving race and performed worse in lower-resource languages such as Tamil and Punjabi.
Featured Image
Photo by Kamil Kalkan on Unsplash
Why is it important?
AI models that read images are being used in news summarization, content moderation, accessibility tools, and crisis response, where mistakes can affect real people. Most existing tests only check whether a model's answer is correct, and a model can score highly while still stereotyping people, overlooking non-English speakers, or failing on imperfect images. HumaniBench is one of the first benchmarks to measure all seven of these human-centred qualities together, using real-world images rather than synthetic ones. Its principles are grounded in governance frameworks such as the EU AI Act and the NIST AI Risk Management Framework, so developers, researchers, and policymakers can check models against the same values regulators care about. The dataset, code, and evaluation tools are freely available, so anyone can test new models and track whether they are becoming more trustworthy over time.
Perspectives
Working on HumaniBench changed how I think about model evaluation. It was easy to assume that a more accurate model is also a better one, but our results kept showing otherwise. Models that topped the accuracy charts could still show clear gaps across racial groups or lose several points when the same question was asked in Tamil instead of English. As someone who speaks a language that AI often underserves, that finding felt personal as well as technical. I'm also proud that we built this on real news imagery with expert human verification, rather than staged or synthetic examples. The work was slower, but the benchmark reflects the messy situations these models actually face. I hope HumaniBench encourages people to ask not just "is the model right?" but also "is it right for everyone, in every language, when it matters?"
Aravind Narayanan
Read the Original
This page is a summary of: HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation, ACM Transactions on Intelligent Systems and Technology, September 2026, ACM (Association for Computing Machinery),
DOI: 10.1145/3845999.
You can read the full text:
Resources
LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation
A companion study that grew out of HumaniBench's multilingual task. It takes a deeper look at how fairly multimodal models answer image-based questions across languages, including low-resource ones.
Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment
A companion study to HumaniBench, accepted at a NeurIPS 2025 workshop. It focuses on social bias: how visible cues such as age, gender, race, and occupation in real news photos shape model answers. It uses 1,343 annotated image–question pairs, with an LLM judge and human checks. Gender and occupation carried the highest bias risk, and models that stayed closer to what's actually in the image didn't always show less bias.
Contributors
The following have contributed to this page
