University of Sao Paolo

Over 35% of patients have at least one allergy recorded in their electronic health records (EHRs), with approximately 4% having three or more documented allergies. Despite this, allergy documentation is often incomplete or inaccurate, impeding effective clinical decision-making and compromising patient safety during prescription [1]. Allergic reactions account for roughly 10% of fatal adverse drug events, underscoring the critical importance of accurate allergy information [2]. Anaphylaxis represents the most severe manifestation of acute systemic allergic reactions, typically occurring within minutes to a few hours following exposure to an allergen or other triggering agent [3].

The accurate documentation and exchange of allergy-related data are essential for ensuring patient safety, effective care delivery, and health education. Research indicates that 8-13% of medication errors could be prevented if allergy information were reliably documented at the time of medication ordering [4]. However, healthcare data are frequently unstructured, inconsistently formatted, or fragmented across different systems, making standardization a complex and resource-intensive process [5, 6].

Recent advancements in deep learning and natural language processing (NLP, particularly through the development of large language models (LLMs), have shown potential for automating the structuring of clinical data at scale. These techniques allow for the extraction of medical entities from unstructured text, the mapping to standardized terminologies such as SNOMED CT, and the generation of structured data in compliance with international healthcare standards, such as FHIR (Fast Healthcare Interoperability Resources) [6, 7].

In this context, we propose a processing pipeline that utilizes LLMs to extract and structure anaphylaxis-related information from free-text clinical notes [11] into the FHIR AllergyIntolerance [10] resource. This pipeline includes mapping of extracted entities, such as substances, clinical manifestations, and exposure routes, to standardized SNOMED CT concepts. To ensure the accuracy and usability of these structured resources, we developed a Web-based application that allows annotators to review and refine the automatically generated data, enhancing the reliability of allergy documentation and reducing the impact of adverse events caused by re-exposure to the identified agents in clinical practice.

The proposed multi-stage processing pipeline begins by extracting clinical entities from unstructured text and structuring them into FHIR-compliant AllergyIntolerance resources in JSON format. To enhance the reliability of this generation process and minimize the occurrence of hallucinations (i.e., the generation of incorrect or extraneous information), structured output generation techniques are employed. These techniques are guided by detailed documentation on the expected output format derived from the official FHIR specification, ensuring that the generated resources conform closely to the standard.

Subsequent to entity extraction, a concept mapping step (also known as entity linking) is performed for those entities requiring binding to standardized medical vocabularies. In particular, entities categorized under substance, manifestation, and exposure route are mapped to corresponding concepts in SNOMED CT. To improve accuracy and contextual relevance, the system utilizes non-parametric external resources, including terminology service APIs. This external access is essential, given that generating correspondences de novo(without additional context) is challenging for large language models [8].

The pipeline operates under a tool-calling agent architecture [9], where each processing stage is modular and can invoke specialized tools as needed. This architecture allows the system to iteratively refine outputs, correcting errors at intermediate steps before they can affect downstream processing. Furthermore, the modular design supports the integration of different models and external tools, enabling comparative evaluation of both open-source and proprietary LLMs.

To assess the pipeline’s performance, system-generated AllergyIntolerance resources are evaluated against expert-annotated datasets. This evaluation considers the accuracy of extracted entities, conformity to the FHIR schema, and correctness of SNOMED CT mappings. For this purpose, a Web-based annotation application was developed, capable of rendering JSON-formatted resources in an intuitive form-based interface. This application enhances usability for clinical and terminological specialists, supports real-time saving, and facilitates collaborative annotation across domains. It also includes author attribution and annotation status tracking features. By streamlining the annotation workflow, this application contributes to the creation of high-quality ground truth datasets, thereby enabling a rigorous evaluation of the pipeline’s effectiveness in automating the conversion of unstructured clinical text into standardized, interoperable formats.

[1] Wang, Liqin et al. A dynamic reaction picklist for improving allergy reaction documentation in the electronic health record. Journal of the American Medical Informatics Association, v. 27, n. 6, p. 917-923, 2020.

[2] Nakayama, Masaharu; Inoue, Ryusuke. Implementation and effect of a novel electronic medical record format for patient allergy information. In: Building Continents of Knowledge in Oceans of Data: The Future of Co-Created eHealth. IOS Press, 2018. p. 51-55.

[3] Cardona V, Ansotegui IJ, Ebisawa M, El-Gamal Y, Fernandez Rivas M, Fineman S, et al. World allergy organization anaphylaxis guidance 2020. World Allergy Organ J. 2020;13(10):100472.

[4] Goss, F. R., Zhou, L., Plasek, J. M., Broverman, C., Robinson, G., Middleton, B., & Rocha, R. A. (2013). Evaluating standard terminologies for encoding allergy information. Journal of the American Medical Informatics Association, 20(5), 969–979.

[5] Mello, Blanda H et al. Semantic interoperability in health records standards: a systematic literature review. In: Health and technology 12.2 (2022), pp. 255–272.

[6] Yang, Xi et al. A large language model for electronic health records. In: NPJ digital medicine 5.1 (2022), p. 194.

[7] Agrawal, Monica et al. Large Language Models are Few-Shot Clinical Information
Extractors. 2022. arXiv: 2205.12689 [cs.CL].

[8] Matentzoglu, Nicolas et al. MapperGPT: Large Language Models for Linking and Mapping Entities. 2023. arXiv: 2310.03666 [cs.CL].

[9] Yao, Shunyu et al. “React: Synergizing reasoning and acting in language models”.

In: arXiv preprint arXiv:2210.03629 (2022).

[10] HL7. Fast Healthcare Interoperability Resources. URL: https://hl7.org/fhir/

[11] Ensina, L.F.; Machado, M.M.; Marques, J.B.M.; dos Santos, M.P.H.; Lario, F.C.; Ara√∫jo, C.A.; Oliveira, F.A.N.; Moreira, D. Artificial intelligence for detecting anaphylaxis in electronic medical records. Asia Pacific Allergy, v. 1, p. 1-6, 2025.