Welcome to Bio-RAMP Lab!

Recent advancements in large language model (LLM) performance on medical multiple-choice question (MCQ) benchmarks have stimulated interest from patients and healthcare providers globally. Particularly in low-and-middle-income countries (LMICs) facing acute physician shortages and lack of specialists, LLMs offer a potentially scalable pathway to enhance healthcare access and reduce costs. However, their effectiveness in the Global South, especially across the African continent, remains to be established due to inherent knowledge gaps. In this work, we introduce AfriMed-QA, the first large-scale multi-specialty Pan-African medical Question-Answer (QA) dataset designed to evaluate and develop equitable and effective LLMs for African healthcare. It contains 3,000 multiple-choice professional medical exam questions with answers and rationale, 1,500 short answer questions (SAQ) with long-form answers, and 5,500 consumer queries (CQ), sourced from over 60 medical schools across 15 countries, covering 32 medical specialties. We further evaluate multiple publicly available general and biomedical LLMs across multiple axes, including accuracy, consistency, bias, potential for harm, local geographic relevance, medical reasoning, and recall. We believe this dataset provides a valuable resource for the practical application of large language models in African healthcare and enhances the geographical diversity of health-LLM benchmark datasets.

However, where named entities such as ”Ukachukwu” (Igbo), ”Lakicia” (Swahili), or ”Ingabire” (Rwandan) are spoken, automatic speech recognition (ASR) models’ performance degrades significantly, propagating errors to downstream systems. We model this problem as a distribution shift and demonstrate that such model bias can be mitigated through multilingual pre-training, intelligent data augmentation strategies to increase the representation of African-named entities, and fine-tuning multilingual ASR models on multi- ple African accents. The resulting fine-tuned models show an 86.4% relative improvement compared with the baseline on samples with African-named entities.