A recent study conducted by investigators from Mass General Brigham reveals that large language models (LLMs), a form of generative artificial intelligence (AI), can be trained to automatically extract information on social determinants of health (SDoH) from doctors’ notes. This breakthrough could significantly enhance efforts to identify patients who may benefit from additional resource support.
The findings, published in npj Digital Medicine, indicate that these finely tuned models were able to identify 93.8 percent of patients with adverse SDoH, whereas official diagnostic codes only included this information in a mere 2 percent of cases. Unlike generalist models such as GPT-4, these specialized models displayed less bias.
Dr. Danielle Bitterman, a faculty member in the Artificial Intelligence in Medicine (AIM) Program at Mass General Brigham and a physician in the Department of Radiation Oncology at Brigham and Women’s Hospital, explained the goal of the study: “Our aim is to identify patients who could benefit from resource and social work support and draw attention to the under-documented impact of social factors on health outcomes.”
It is widely acknowledged that health disparities often stem from SDoH, encompassing non-medical factors such as employment, housing, and other circumstances that can impact medical care. For instance, the proximity of a cancer patient’s residence to a major medical center or the level of support they receive from a partner can significantly affect treatment outcomes. While clinicians may mention pertinent SDoH in their visit notes, this vital information is seldom systematically organized in electronic health records (EHRs).
To develop language models capable of extracting SDoH information, the researchers manually reviewed 800 clinician notes from 770 cancer patients who received radiotherapy at Brigham and Women’s Hospital. They identified sentences pertaining to six predetermined SDoH: employment status, housing, transportation, parental status, relationships, and social support.
Using this annotated dataset, the researchers trained existing language models to recognize references to SDoH in doctors’ notes. They subsequently tested their models using 400 clinic notes from patients undergoing immunotherapy at Dana-Farber Cancer Institute and those admitted to critical care units at Beth Israel Deaconess Medical Center.
The study revealed that fine-tuned language models, particularly Flan-T5 models, consistently recognized rare mentions of SDoH in doctors’ notes. However, the researchers encountered limitations due to the scarcity of SDoH documentation in the training set. Only 3 percent of sentences in clinician notes contained any reference to SDoH. To address this issue, the researchers generated an additional 900 synthetic examples of SDoH sentences using ChatGPT, another language model, to serve as supplementary training data.
One major concern with generative AI models in healthcare is the potential to perpetuate bias and exacerbate health disparities. In this study, the fine-tuned language model demonstrated a lower likelihood of changing its determination about an SDoH based on individuals’ race/ethnicity and gender compared to OpenAI’s GPT-4, a generalist language model.
The researchers acknowledge that understanding the formation and deconstruction of biases in both human and computer models is a complex task. Identifying algorithmic biases remains an ongoing pursuit for the researchers.
Dr. Bitterman emphasized the importance of monitoring algorithmic bias during the development and implementation of large language models: “If we don’t address algorithmic bias, we risk worsening existing health disparities. This study shows that fine-tuning language models may be a strategy to mitigate algorithmic bias, but further research in this area is necessary.”
Ravina Pandya, Content Writer, has a strong foothold in the market research industry. She specializes in writing well-researched articles from different industries, including food and beverages, information and technology, healthcare, chemical and materials, etc. With an MBA in E-commerce, she has an expertise in SEO-optimized content that resonates with industry professionals.