Centre for the Fourth Industrial Revolution Network 2023-2024

18 Action: The participants running the pilot developed a tool that leverages GPT-4 as a knowledge base and uses an Mbaza English-Kinyarwanda translation model to empower CHWs in Rwanda. This pilot focused specifically on malnutrition, which remains a critical challenge in Rwanda.Case study 5: LLM tool for community health workers Rwanda Context: In Rwanda, community health workers (CHWs) are the backbone of primary healthcare. Yet, CHWs’ lack of clinical training leads to delayed diagnoses, preventable deaths and extreme strain on the public health system. Large language models (LLMs) are showing significant potential in closing knowledge gaps in healthcare, but for this to be deployed in Rwanda, LLMs must converse in Kinyarwanda. Result: Preliminary assessments have yielded promising results, underscoring the potential of LLMs to drive positive change in Rwanda’s healthcare landscape. Figure 1: Network outcomes Baseline 8% accuracy, as evaluated by professional linguists (that is, it did not comprehend Kinyarwanda at all; did not evaluate with healthcare workers).55% accuracy, as evaluated by professional linguists To note: GPT-4’s comprehension of Kinyarwanda has improved dramatically over the last three months. Open-source models still lack comprehension.71% functional accuracy, as evaluated by professional healthcare workers (that is, responses are comprehensible but have grammatical errors). After pilot Next steps: The next phase of rolling out this LLM tool includes two steps: 1 Evaluation of LLM datasets by collecting questions from CHWs and responses from both clinicians and LLMs. This dataset will be used to evaluate the performance of LLMs in a clinical setting.2 Conducting a silent trial of an AI-enhanced CHW app in Rwanda, with a primary focus on evaluating the performance of LLMs within the CHW setting. The trial will assess the LLMs’ ability to provide accurate and culturally relevant decision support for diagnosis and referral without altering CHW behaviour.