Evaluating Contextual Embedding Models for Multi-Label PICO Classification in Heart Disease: Addressing the Intervention - Comparison Bottleneck
DOI:
https://doi.org/10.31849/digitalzone.v16i2.28375Keywords:
PICO classification, BioBERT, PubMedBERT, SciBERT, BiLSTM, clinical NLP, heart disease, contextual embedding, evidence-based medicineAbstract
Accurate extraction of Population, Intervention, Comparison, and Outcome (PICO) elements from clinical texts is essential for supporting evidence-based medicine, particularly in cardiology where clinical data complexity presents significant challenges. This study investigates the comparative effectiveness of three contextual embedding models—BioBERT, PubMedBERT, and SciBERT—integrated with a Bidirectional Long Short-Term Memory (BiLSTM) architecture for multi-label PICO element classification on heart disease datasets. The experimental framework involved pre-processing clinical sentences, transforming them into contextual embeddings, and classifying PICO elements using BiLSTM-based sequence modeling. Evaluation was conducted using five key metrics: accuracy, precision, recall, F1-score, and hamming loss, supplemented by confusion matrix analysis for each PICO element. Results demonstrate that the BioBERT-BiLSTM model achieved superior performance, with an accuracy of 73.89%, F1-score of 78.54%, precision of 81.60%, and recall of 76.64%. PubMedBERT-BiLSTM exhibited the highest precision (84.12%) but lower recall, while SciBERT-BiLSTM produced slightly inferior results overall. These findings confirm the importance of using domain-specific embeddings, particularly models pre-trained on biomedical corpora, to improve classification accuracy in specialized clinical text tasks. This study concludes that the BioBERT-BiLSTM combination offers a reliable approach for automated PICO element extraction in the cardiology domain, contributing to the development of more accurate and efficient clinical decision-support systems
References
[1] A. Ramic-Catak, S. Mesihovic-Dinarevic, B. Prnjavorac, N. Naser, and I. Masic, “Public Health Dimensions of Cardiovascular Diseases (CVD) Prevention and Control – Global Perspectives and Current Situation in the Federation of Bosnia and Herzegovina,” Mater. Sociomed., vol. 35, no. 2, pp. 88–93, 2023, https://doi.org/10.5455/msm.2023.35.88-93.
[2] J. J. Cárdenas-Anguiano et al., “Estimation of the Burden of Ischemic Heart Disease in the Tabasco Population, Mexico, 2013–2021,” 2025. https://doi.org/10.3390/ijerph22030423.
[3] M. Borges et al., “Burden of Disease and Cost of Illness of Overweight and Obesity in Portugal,” 2024. doi: 10.1159/000541781.
[4] B. Sakboonyarat and R. Rangsin, “Hospital Admission and Mortality Rates for Ischemic Heart Disease in Thailand: 2012–2021,” 2024. https://doi.org/10.1186/s13104-024-06803-x.
[5] A. K. Nagpal, A. Pundkar, A. Singh, and C. Gadkari, “Cardiac Arrhythmias and Their Management: An in-Depth Review of Current Practices and Emerging Therapies,” 2024.https://doi.org/10.7759/cureus.66549.
[6] A. Sharma, “Incidental Coronary Artery Disease on Routine CT Coronary Angiography –An Evidence-Based Approach,” 2023. https://doi.org/10.58489/2836-5917/005.
[7] R. Javid et al., “Transcoronary Electrophysiological Parameters in Patients Undergoing Elective and Acute Coronary Intervention,” 2023. https://doi.org/10.1371/journal.pone.0281374.
[8] V. Kittipibul et al., “Projecting the Benefit of Vericiguat in PARADIGM‐HF and DAPA‐HF Populations: Insights From the VICTORIA Trial,” 2024. https://doi.org/10.1002/ehf2.15134.
[9] X. Chen, M. Zeng, C. Chen, D. Zhu, L. Chen, and Z. Jiang, “Efficacy of Psycho-Cardiology Therapy in Patients With Acute Myocardial Infarction Complicated With Mild Anxiety and Depression,” 2023. https://doi.org/10.3389/fcvm.2022.1031255.
[10] A. Vahanian et al., “2021 ESC/EACTS Guidelines for the Management of Valvular Heart Disease,” 2022. doi: 10.4244/eij-e-21-00009.
[11] E. Hiraoka et al., “JCS 2022 Guideline on Perioperative Cardiovascular Assessment and Management for Non-Cardiac Surgery,” 2023. https://doi.org/10.1253/circj.cj-22-0609.
[12] D. Kartchner et al., “TrialSieve: A Comprehensive Biomedical Information Extraction Framework for PICO, Meta-Analysis, and Drug Repurposing,” 2025. https://doi.org/10.3390/bioengineering12050486.
[13] A. Ragnhildstveit et al., “Intra-Operative Applications of Augmented Reality in Glioma Surgery: A Systematic Review,” 2023. https://doi.org/10.3389/fsurg.2023.1245851.
[14] A. M. Ngo, A. Donaghue, K. H. Weng, and E. T. Crehan, “Barriers and Facilitators for Addressing Sex Education for Autistic Individuals: A Systematic Review,” 2024. https://doi.org/10.1177/00224669241292045.
[15] M. Cumpston, J. E. McKenzie, V. Welch, and S. Brennan, “Strengthening Systematic Reviews in Public Health: Guidance in the Cochrane Handbook for Systematic Reviews of Interventions, 2nd Edition,” 2022. https://doi.org/10.1093/pubmed/fdac036.
[16] C. Witte, D. Schmidt, and P. Cimiano, “Comparing Generative and Extractive Approaches to Information Extraction From Abstracts Describing Randomized Clinical Trials,” 2024. https://doi.org/10.1186/s13326-024-00305-2.
[17] D. C. Moore and A. S. Guinigundo, “Biomarker-Driven Oncology Clinical Trials: Novel Designs in the Era of Precision Medicine,” 2023. https://doi.org/10.6004/jadpro.2023.14.3.16.
[18] B. P. Carlin and F. Nollevaux, “Bayesian Complex Innovative Trial Designs (CIDs) and Their Use in Drug Development for Rare Disease,” 2022. https://doi.org/10.1002/jcph.2132.
[19] A. Griessbach et al., “Characteristics, Progression, and Output of Randomized Platform Trials,” 2024. https://doi.org/10.1001/jamanetworkopen.2024.3109.
[20] F. Gay et al., “Clinical Outcomes Associated With Anti-Cd38-Based Retreatment in Relapsed/Refractory Multiple Myeloma: A Systematic Literature Review,” 2025. https://doi.org/10.3389/fonc.2025.1550644.
[21] L. A. Nkhata, A. Human, Q. Louw, and Y. Brink, “Psychometric Properties and Clinical Utility of Spinal Health Outcome Measures in School-Based Interventions Among Children and Adolescents: A Systematic Review Protocol,” 2024. https://doi.org/10.1136/bmjopen-2024-089929.
[22] L. E. Almeida, S. Zammuto, and D. F. López, “Evaluating Surgical Approaches for Hemimandibular Hyperplasia Associated With Osteochondroma: A Systematic Literature Review,” 2024. https://doi.org/10.3390/jcm13226988.
[23] Y. Zang et al., “Evidence Mapping Based on Systematic Reviews of Repetitive Transcranial Magnetic Stimulation on the Motor Cortex for Neuropathic Pain,” 2022. https://doi.org/10.3389/fnhum.2021.743846.
[24] M. Cumpston, J. E. McKenzie, R. Ryan, E. Flemyng, J. Thomas, and S. Brennan, “Development of the InSynQ Checklist: A Tool for Planning and Reporting the Synthesis Questions in Systematic Reviews of Interventions,” 2023. https://doi.org/10.1002/cesm.12036.
[25] Q. Wang, “PICO entity extraction for preclinical animal literature,” Syst. Rev., vol. 11, no. 1, 2022, https://doi.org/10.1186/s13643-022-02074-4.
[26] O. Sanchez-Graillet, C. Witte, F. Grimm, and P. Cimiano, “An Annotated Corpus of Clinical Trial Publications Supporting Schema-Based Relational Information Extraction,” 2022. https://doi.org/10.1186/s13326-022-00271-7.
[27] A. Dhrangadhariya and H. Müller, “Not So Weak PICO: Leveraging Weak Supervision for Participants, Interventions, and Outcomes Recognition for Systematic Review Automation,” 2023. https://doi.org/ 10.1093/jamiaopen/ooac107.
[28] A. Jolly, V. Pandey, I. Singh, and N. Sharma, “Exploring Biomedical Named Entity Recognition via SciSpaCy and BioBERT Models,” Open Biomed. Eng. J., vol. 18, no. 1, 2024, https://doi.org/10.2174/0118741207289680240510045617.
[29] V. Dóczy, B. W. Sódar, Á. Hölgyesi, G. Merész, and P. Gaál, “Development, Testing, and Implementation of a New Procedure to Assess the Clinical Added Benefit of Pharmaceuticals,” 2022. https://doi.org/10.1017/s0266462322000411.
[30] K. Eberle et al., “The PICO Puzzle: Can Public Data Predict EU HTA Expectations for All EU Countries?,” 2025. https://doi.org/10.3390/jmahp13030032.
[31] O. Ahmad, “PICO medical literatures on heart disease data,” 2021. Accessed: Jan. 17, 2025. [Online]. Available: https://www.kaggle.com/datasets/owaiskhan9654/pico-medical-literatures-related-to-heart-disease
[32] Y. Gu et al., “Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing,” ACM Trans. Comput. Healthc., vol. 3, no. 1, Oct. 2021, https://doi.org/10.1145/3458754.
[33] I. Beltagy, K. Lo, and A. Cohan, “{S}ci{BERT}: A Pretrained Language Model for Scientific Text,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan, Eds., Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3615–3620. https://doi.org/10.18653/v1/D19-1371.
[34] J. Li et al., “A comparative study of pre-trained language models for named entity recognition in clinical trial eligibility criteria from multiple corpora,” BMC Med. Inform. Decis. Mak., vol. 22, no. Suppl 3, p. 235, 2022, https://doi.org/10.1186/s12911-022-01967-7.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Digital Zone: Jurnal Teknologi Informasi dan Komunikasi
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Jurnal Digital Zone is licensed under aCreative Commons Attribution-ShareAlike 4.0 International License.
