--- title: Natural Language Processing for Medical Report Analysis - Part 2 date: 2026-05-09T14:52:29Z modified: 2026-05-25T06:23:28Z permalink: "https://www.micheledpierri.com/2026/05/09/advanced-spacy-scispacy-clinical-nlp/" type: post status: publish excerpt: "" wpid: 2692 categories: - Health Informatics - Medicine tags: - Health Informatics - Medicine - Data Science - NLP - Python featured_image: "https://www.micheledpierri.com/wp-content/uploads/2026/05/NLP2.png" featured_image_alt: A doctor translates text timestamp: 2026-05-25T06:23:28Z --- # Introduction to Natural Language Processing for Medical Report Analysis: Advanced Linguistic Analysis with spaCy and scispaCy for Medical Reports --- _Article authored by Michele D. Pierri, MD_ _Cardiac Surgeon & Medical Technology Researcher_ _Last updated: May 2025_ _Reading time: 30 minutes_ --- ## Abstract Regular expressions give us a solid foundation for pulling structured data out of clinical documents. They struggle, though, with the semantic complexity that defines medical language. This article introduces two complementary libraries (spaCy and scispaCy) that bring genuine linguistic intelligence to clinical text. We work through detailed examples on a post-CABG discharge summary, covering tokenization, part-of-speech tagging, dependency parsing, and Named Entity Recognition (NER) with biomedical models. UMLS entity linking is also explored, since it ties extracted clinical terms to standardized medical ontologies and enables interoperability across health information systems. By the final section the reader will know not only how to implement these techniques in Python, but when to prefer them over regex; and when, in the daily reality of clinical NLP, the two should be combined. **Article Series:** 1. [Introduction to NLP and Regex for Medical Reports](https://www.micheledpierri.com/wp-content/uploads/wp-mfa-exports/post/natural-language-processing-for-medical-report.md) 2. **Advanced Linguistic Analysis with spaCy and scispaCy** 3. [Machine Learning and Large Language Models for Clinical Text Analysis](https://www.micheledpierri.com/wp-content/uploads/wp-mfa-exports/post/machine-learning-and-large-language-models-for-clinical-text-analysis.md) --- ## 1. Beyond Pattern Matching: Why Linguistic Analysis? ### 1.1 The Limits of Regular Expressions The first article of this series built a regex parser able to extract patient demographics, surgical parameters, medications, and complications from a post-CABG discharge summary. That implementation worked because our sample document followed predictable formatting conventions. Real-world clinical documentation, in our experience reading hundreds of operative reports across centers, is far less disciplined. Consider these semantically equivalent phrases that all describe the same complication: "Postoperative atrial fibrillation was noted on POD 3" "Patient developed new-onset AF in the immediate post-surgical period" "Rhythm monitoring revealed paroxysmal atrial fibrillation 72 hours after bypass" "POD 3: AF, cardioverted successfully" ``` "Postoperative atrial fibrillation was noted on POD 3" "Patient developed new-onset AF in the immediate post-surgical period" "Rhythm monitoring revealed paroxysmal atrial fibrillation 72 hours after bypass" "POD 3: AF, cardioverted successfully" ``` A regex pattern designed to catch the first phrasing will miss the others. We could write extra patterns for each variant, but this becomes an arms race against the variability of natural language. Regex also cannot: - Determine that “no wound infections” and “wound infection noted” have opposite clinical meanings (negation handling) - Understand that “CABG,” “bypass surgery,” and “coronary revascularization” refer to the same procedure - Extract the _relationship_ between a drug and its indication (“amiodarone for atrial fibrillation”) - Handle abbreviation disambiguation (“AF” could mean atrial fibrillation or aortic flow depending on context) These limitations are what push us toward NLP tools built on computational linguistics, the scientific study of language structure. ### 1.2 The Linguistic Approach Natural language processing at the linguistic level rests on a simple premise: language has internal structure, and that structure can be computationally modeled. Rather than matching character sequences, linguistic NLP: 1. **Tokenizes** text into meaningful units (words, punctuation, medical abbreviations) 2. **Tags** each token with its grammatical role (noun, verb, adjective) 3. **Parses** the syntactic relationships between tokens (subject, object, modifier) 4. **Recognizes** named entities, that is, spans of text referring to real-world concepts (diseases, drugs, procedures) 5. **Links** recognized entities to external knowledge bases (UMLS, SNOMED-CT, RxNorm) The pipeline turns raw text not into matched strings, but into a structured representation of linguistic meaning. ### 1.3 The spaCy Ecosystem for Biomedical Text spaCy is an industrial-strength NLP library for Python, designed for production use rather than academic experimentation. It delivers fast, accurate linguistic analysis through pre-trained statistical models built on large corpora. For general English text, the `en_core_web_sm/md/lg` models are the standard starting point. Medical language presents a domain-specific challenge. General-purpose NLP models are trained predominantly on news articles, books, and web content, corpora that share little with clinical documentation. Medical text is characterized by: - High density of Latin and Greek terminology - Extensive use of abbreviations and acronyms (POD, CABG, LVEF, LIMA) - Domain-specific syntactic patterns - Specialized named entity types (diseases, procedures, anatomical structures, medications) - Negation patterns with clinical significance (“no evidence of”, “without complications”) This is where **scispaCy** earns its place. Developed by the Allen Institute for AI, scispaCy provides spaCy-compatible models trained on biomedical scientific literature. The performance gap on clinical text, compared to general-purpose models, is substantial. --- ## 2. spaCy Fundamentals for Medical Text ### 2.1 Installation and Setup Before proceeding, ensure you have the required libraries installed. spaCy and scispaCy require careful version management: # Install spaCy pip install spacy # Install scispaCy pip install scispacy # Install a scispaCy biomedical model # en_core_sci_md offers a good balance of accuracy and performance pip install # For UMLS entity linking (requires additional setup) pip install scispacy[linker] ``` # Install spaCy pip install spacy # Install scispaCy pip install scispacy # Install a scispaCy biomedical model # en_core_sci_md offers a good balance of accuracy and performance pip install <https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_md-0.5.4.tar.gz> # For UMLS entity linking (requires additional setup) pip install scispacy[linker] ``` > **Note on model selection**: scispaCy provides several pre-trained models. `en_core_sci_sm` is the smallest and fastest; `en_core_sci_md` includes word vectors for better similarity comparisons; `en_core_sci_lg` offers maximum accuracy. For production medical applications, `en_core_sci_md` or `en_core_sci_lg` is recommended. For research requiring UMLS linking specifically, `en_core_sci_lg` combined with the `UmlsEntityLinker` is the gold standard. ### 2.2 The spaCy Document Object Model When spaCy processes a text string, it returns a `Doc` object: a rich data structure that exposes all linguistic annotations. Understanding this object model is essential before doing anything serious with the library. import spacy # Load the general English model first to understand the basics # (we will switch to scispaCy models for medical text shortly) nlp = spacy.load("en_core_web_sm") sample_text = """ The patient was extubated on POD 1 without complications. Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone loading followed by maintenance dose. """ doc = nlp(sample_text) # The Doc object contains Token objects print(f"Number of tokens:{len(doc)}") # Each token exposes multiple attributes for token in doc[:10]: # First 10 tokens print(f" Token:{token.text!r:20} | POS:{token.pos_:10} | Lemma:{token.lemma_:20} | Stop:{token.is_stop}") ``` import spacy # Load the general English model first to understand the basics # (we will switch to scispaCy models for medical text shortly) nlp = spacy.load("en_core_web_sm") sample_text = """ The patient was extubated on POD 1 without complications. Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone loading followed by maintenance dose. """ doc = nlp(sample_text) # The Doc object contains Token objects print(f"Number of tokens:{len(doc)}") # Each token exposes multiple attributes for token in doc[:10]: # First 10 tokens print(f" Token:{token.text!r:20} | POS:{token.pos_:10} | Lemma:{token.lemma_:20} | Stop:{token.is_stop}") ``` Output: Number of tokens: 40 Token: ‘The’ | POS: DET | Lemma: ‘the’ | Stop: True Token: ‘patient’ | POS: NOUN | Lemma: ‘patient’ | Stop: False Token: ‘was’ | POS: AUX | Lemma: ‘be’ | Stop: True Token: ‘extubated’ | POS: VERB | Lemma: ‘extubate’ | Stop: False Token: ‘on’ | POS: ADP | Lemma: ‘on’ | Stop: True Token: ‘POD’ | POS: PROPN | Lemma: ‘POD’ | Stop: False Token: ‘1’ | POS: NUM | Lemma: ‘1’ | Stop: False ### 2.3 Tokenization in Medical Text Tokenization (splitting text into individual tokens) sounds trivial, but in clinical NLP it is anything but. Take “Amiodarone 200mg”: should this be one token or two? What about “POD 3”, or “LVEF 40%”? Medical abbreviations punctuated with periods (“M.D.”, “i.v.”) add another layer of trouble. spaCy’s tokenization rules are based on the Penn Treebank standard but can be customized. Let us look at how tokenization handles our discharge summary: import spacy from typing import List, Dict def analyze_tokenization(text: str, nlp) -> List[Dict]: """ Analyze how spaCy tokenizes medical text. This function reveals tokenization decisions that are clinically significant - for example, whether "100mg" is treated as one token or split into "100" and "mg". Args: text: Clinical text to tokenize nlp: Loaded spaCy model Returns: List of token dictionaries with linguistic attributes """ doc = nlp(text) tokens = [] for token in doc: tokens.append({ 'text': token.text, 'lemma': token.lemma_, 'pos': token.pos_, # Coarse-grained POS (NOUN, VERB, etc.) 'tag': token.tag_, # Fine-grained POS (NNS, VBD, etc.) 'dep': token.dep_, # Syntactic dependency role 'is_alpha': token.is_alpha, 'is_digit': token.is_digit, 'is_stop': token.is_stop, # Common words with little semantic value 'is_punct': token.is_punct, 'shape': token.shape_, # Character shape: "100mg" → "dddxx" }) return tokens # Test on the medication section of our discharge summary medication_text = """ DISCHARGE MEDICATIONS: 1. Aspirin 100mg daily 2. Clopidogrel 75mg daily (for 12 months) 3. Atorvastatin 80mg daily 4. Metoprolol 50mg twice daily 5. Ramipril 5mg daily 6. Amiodarone 200mg daily (for 6 weeks) """ # We'll use scispaCy for actual medical processing nlp_sci = spacy.load("en_core_sci_md") tokens = analyze_tokenization(medication_text, nlp_sci) # Print clinically relevant tokens (non-stop, non-punctuation) print("Clinically Relevant Tokens:") print(f"{'Text':<20}{'Lemma':<20}{'POS':<10}{'Shape':<15}") print("-" * 70) for t in tokens: if not t['is_stop'] and not t['is_punct'] and t['text'].strip(): print(f"{t['text']:<20}{t['lemma']:<20}{t['pos']:<10}{t['shape']:<15}") ``` import spacy from typing import List, Dict def analyze_tokenization(text: str, nlp) -> List[Dict]: """ Analyze how spaCy tokenizes medical text. This function reveals tokenization decisions that are clinically significant - for example, whether "100mg" is treated as one token or split into "100" and "mg". Args: text: Clinical text to tokenize nlp: Loaded spaCy model Returns: List of token dictionaries with linguistic attributes """ doc = nlp(text) tokens = [] for token in doc: tokens.append({ 'text': token.text, 'lemma': token.lemma_, 'pos': token.pos_, # Coarse-grained POS (NOUN, VERB, etc.) 'tag': token.tag_, # Fine-grained POS (NNS, VBD, etc.) 'dep': token.dep_, # Syntactic dependency role 'is_alpha': token.is_alpha, 'is_digit': token.is_digit, 'is_stop': token.is_stop, # Common words with little semantic value 'is_punct': token.is_punct, 'shape': token.shape_, # Character shape: "100mg" → "dddxx" }) return tokens # Test on the medication section of our discharge summary medication_text = """ DISCHARGE MEDICATIONS: 1. Aspirin 100mg daily 2. Clopidogrel 75mg daily (for 12 months) 3. Atorvastatin 80mg daily 4. Metoprolol 50mg twice daily 5. Ramipril 5mg daily 6. Amiodarone 200mg daily (for 6 weeks) """ # We'll use scispaCy for actual medical processing nlp_sci = spacy.load("en_core_sci_md") tokens = analyze_tokenization(medication_text, nlp_sci) # Print clinically relevant tokens (non-stop, non-punctuation) print("Clinically Relevant Tokens:") print(f"{'Text':<20}{'Lemma':<20}{'POS':<10}{'Shape':<15}") print("-" * 70) for t in tokens: if not t['is_stop'] and not t['is_punct'] and t['text'].strip(): print(f"{t['text']:<20}{t['lemma']:<20}{t['pos']:<10}{t['shape']:<15}") ``` **Clinical Insight on Tokenization**: scispaCy handles medical abbreviations more gracefully than general models, although the picture is not uniform. “100mg” may be split or kept together depending on the model and configuration. For downstream extraction tasks, custom tokenization rules for domain-specific patterns are often beneficial: from spacy.lang.char_classes import ALPHA, ALPHA_LOWER, ALPHA_UPPER from spacy.lang.en import English def add_medical_tokenization_rules(nlp): """ Add custom tokenization rules for medical text. Medical documents contain patterns that standard tokenizers handle poorly: drug-dose combinations, medical acronyms, and clinical measurement notations. """ # Prevent splitting on medical dose patterns like "100mg", "5mg" # These should remain as single tokens for accurate extraction infixes = nlp.Defaults.infixes # Add special cases for common medical abbreviations special_cases = { "POD": [{"ORTH": "POD"}], # Postoperative Day "CPB": [{"ORTH": "CPB"}], # Cardiopulmonary Bypass "CABG": [{"ORTH": "CABG"}], # Coronary Artery Bypass Grafting "LIMA": [{"ORTH": "LIMA"}], # Left Internal Mammary Artery "SVG": [{"ORTH": "SVG"}], # Saphenous Vein Graft "LVEF": [{"ORTH": "LVEF"}], # Left Ventricular Ejection Fraction "LAD": [{"ORTH": "LAD"}], # Left Anterior Descending "RCA": [{"ORTH": "RCA"}], # Right Coronary Artery } for text, case in special_cases.items(): nlp.tokenizer.add_special_case(text, case) return nlp ``` from spacy.lang.char_classes import ALPHA, ALPHA_LOWER, ALPHA_UPPER from spacy.lang.en import English def add_medical_tokenization_rules(nlp): """ Add custom tokenization rules for medical text. Medical documents contain patterns that standard tokenizers handle poorly: drug-dose combinations, medical acronyms, and clinical measurement notations. """ # Prevent splitting on medical dose patterns like "100mg", "5mg" # These should remain as single tokens for accurate extraction infixes = nlp.Defaults.infixes # Add special cases for common medical abbreviations special_cases = { "POD": [{"ORTH": "POD"}], # Postoperative Day "CPB": [{"ORTH": "CPB"}], # Cardiopulmonary Bypass "CABG": [{"ORTH": "CABG"}], # Coronary Artery Bypass Grafting "LIMA": [{"ORTH": "LIMA"}], # Left Internal Mammary Artery "SVG": [{"ORTH": "SVG"}], # Saphenous Vein Graft "LVEF": [{"ORTH": "LVEF"}], # Left Ventricular Ejection Fraction "LAD": [{"ORTH": "LAD"}], # Left Anterior Descending "RCA": [{"ORTH": "RCA"}], # Right Coronary Artery } for text, case in special_cases.items(): nlp.tokenizer.add_special_case(text, case) return nlp ``` ### 2.4 Part-of-Speech Tagging Part-of-speech (POS) tagging assigns each token its grammatical role. In medical NLP, POS tags are useful for identifying: - **Nouns** (NOUN, PROPN): Disease names, anatomical structures, drugs - **Verbs** (VERB): Procedures performed, clinical events (“extubated,” “cardioverted”) - **Adjectives** (ADJ): Qualitative descriptions (“satisfactory,” “stable,” “reduced”) - **Numbers** (NUM): Dosages, durations, laboratory values def extract_clinical_verbs(text: str, nlp) -> List[Dict]: """ Extract action verbs from clinical text. In cardiac surgery reports, verbs often encode critical procedural and clinical events: "performed," "extubated," "cardioverted," "noted," "removed." This function identifies these for downstream event extraction. Args: text: Clinical narrative text nlp: scispaCy model Returns: List of clinical verbs with their lemmas and surrounding context """ doc = nlp(text) clinical_verbs = [] for token in doc: if token.pos_ == "VERB" and not token.is_stop: # Get 3-word context window around verb start = max(0, token.i - 3) end = min(len(doc), token.i + 4) context = doc[start:end].text clinical_verbs.append({ 'verb': token.text, 'lemma': token.lemma_, 'tense': token.tag_, # VBD = past tense, VBN = past participle 'context': context, 'negated': _check_negation(token, doc) # See negation section }) return clinical_verbs def _check_negation(token, doc) -> bool: """ Simple negation detection using dependency parsing. Checks if a token has a negation modifier in its dependency tree. More sophisticated negation handling (like NegEx algorithm) is covered in the discussion section. """ for child in token.children: if child.dep_ == "neg": return True return False # Apply to the postoperative course section postop_text = """ The patient was extubated on POD 1 without complications. Chest tubes were removed on POD 2. Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone loading. No wound infections or sternal instability noted. """ nlp_sci = spacy.load("en_core_sci_md") verbs = extract_clinical_verbs(postop_text, nlp_sci) print("Clinical Verbs Detected:") print(f"{'Verb':<20}{'Lemma':<20}{'Tense':<8}{'Negated':<10}{'Context'}") print("-" * 90) for v in verbs: print(f"{v['verb']:<20}{v['lemma']:<20}{v['tense']:<8}{str(v['negated']):<10}{v['context']}") ``` def extract_clinical_verbs(text: str, nlp) -> List[Dict]: """ Extract action verbs from clinical text. In cardiac surgery reports, verbs often encode critical procedural and clinical events: "performed," "extubated," "cardioverted," "noted," "removed." This function identifies these for downstream event extraction. Args: text: Clinical narrative text nlp: scispaCy model Returns: List of clinical verbs with their lemmas and surrounding context """ doc = nlp(text) clinical_verbs = [] for token in doc: if token.pos_ == "VERB" and not token.is_stop: # Get 3-word context window around verb start = max(0, token.i - 3) end = min(len(doc), token.i + 4) context = doc[start:end].text clinical_verbs.append({ 'verb': token.text, 'lemma': token.lemma_, 'tense': token.tag_, # VBD = past tense, VBN = past participle 'context': context, 'negated': _check_negation(token, doc) # See negation section }) return clinical_verbs def _check_negation(token, doc) -> bool: """ Simple negation detection using dependency parsing. Checks if a token has a negation modifier in its dependency tree. More sophisticated negation handling (like NegEx algorithm) is covered in the discussion section. """ for child in token.children: if child.dep_ == "neg": return True return False # Apply to the postoperative course section postop_text = """ The patient was extubated on POD 1 without complications. Chest tubes were removed on POD 2. Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone loading. No wound infections or sternal instability noted. """ nlp_sci = spacy.load("en_core_sci_md") verbs = extract_clinical_verbs(postop_text, nlp_sci) print("Clinical Verbs Detected:") print(f"{'Verb':<20}{'Lemma':<20}{'Tense':<8}{'Negated':<10}{'Context'}") print("-" * 90) for v in verbs: print(f"{v['verb']:<20}{v['lemma']:<20}{v['tense']:<8}{str(v['negated']):<10}{v['context']}") ``` ### 2.5 Dependency Parsing: Understanding Relationships Dependency parsing is where linguistic analysis starts to look like genuine clinical intelligence. A dependency parser builds a tree showing how words in a sentence relate to each other grammatically. In medical text, this opens the door to extracting structured relationships: - _Subject_ → _Verb_ → _Object_: “amiodarone cardioverted atrial fibrillation” - _Modifier_ → _Head noun_: “postoperative atrial fibrillation” (adjective modifying noun) - _Prepositional_ relationships: “cardioverted _with_ amiodarone” (instrument of action) def extract_drug_event_relations(text: str, nlp) -> List[Dict]: """ Extract drug-event relationships using dependency parsing. Dependency parsing allows us to identify sentences like: "successfully cardioverted with amiodarone" → drug: amiodarone, action: cardioverted, outcome: successful This is a level of extraction impossible with regex alone, as it requires understanding sentence structure, not just character patterns. Args: text: Clinical text containing drug and event mentions nlp: Loaded spaCy/scispaCy model Returns: List of extracted drug-event relationships """ doc = nlp(text) relations = [] # Medical procedure verbs relevant to cardiac surgery PROCEDURE_VERBS = { 'cardiovert', 'extubate', 'remove', 'perform', 'complete', 'administer', 'initiate', 'discontinue' } # Common cardiac drug keywords CARDIAC_DRUGS = { 'amiodarone', 'aspirin', 'clopidogrel', 'atorvastatin', 'metoprolol', 'ramipril', 'furosemide', 'metformin' } for token in doc: # Find procedure verbs if token.lemma_.lower() in PROCEDURE_VERBS: relation = { 'verb': token.text, 'verb_lemma': token.lemma_, 'drug': None, 'patient_subject': False, 'outcome_modifier': None, 'sentence': token.sent.text.strip() } # Traverse dependency tree for related tokens for child in token.children: # Check for drug as instrument (prepositional: "with amiodarone") if child.dep_ == "prep": for grandchild in child.children: if grandchild.lemma_.lower() in CARDIAC_DRUGS: relation['drug'] = grandchild.text # Check for patient as subject ("patient was extubated") if child.dep_ in ("nsubj", "nsubjpass"): if child.lemma_.lower() in ("patient",): relation['patient_subject'] = True # Check for outcome adverbs ("successfully cardioverted") if child.dep_ == "advmod": relation['outcome_modifier'] = child.text # Also check the token's own head for drugs for sibling in token.head.children: if sibling.lemma_.lower() in CARDIAC_DRUGS and sibling != token: if not relation['drug']: relation['drug'] = sibling.text if relation['verb']: relations.append(relation) return relations # Test on the postoperative section postop_text = """ The patient was extubated on POD 1 without complications. Chest tubes were removed on POD 2. Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone loading followed by maintenance dose. """ relations = extract_drug_event_relations(postop_text, nlp_sci) print("Drug-Event Relationships Extracted:") for r in relations: print(f"\\n Verb:{r['verb']}") print(f" Drug:{r['drug'] or 'N/A'}") print(f" Patient subject:{r['patient_subject']}") print(f" Outcome modifier:{r['outcome_modifier'] or 'N/A'}") print(f" Sentence:{r['sentence'][:80]}...") ``` def extract_drug_event_relations(text: str, nlp) -> List[Dict]: """ Extract drug-event relationships using dependency parsing. Dependency parsing allows us to identify sentences like: "successfully cardioverted with amiodarone" → drug: amiodarone, action: cardioverted, outcome: successful This is a level of extraction impossible with regex alone, as it requires understanding sentence structure, not just character patterns. Args: text: Clinical text containing drug and event mentions nlp: Loaded spaCy/scispaCy model Returns: List of extracted drug-event relationships """ doc = nlp(text) relations = [] # Medical procedure verbs relevant to cardiac surgery PROCEDURE_VERBS = { 'cardiovert', 'extubate', 'remove', 'perform', 'complete', 'administer', 'initiate', 'discontinue' } # Common cardiac drug keywords CARDIAC_DRUGS = { 'amiodarone', 'aspirin', 'clopidogrel', 'atorvastatin', 'metoprolol', 'ramipril', 'furosemide', 'metformin' } for token in doc: # Find procedure verbs if token.lemma_.lower() in PROCEDURE_VERBS: relation = { 'verb': token.text, 'verb_lemma': token.lemma_, 'drug': None, 'patient_subject': False, 'outcome_modifier': None, 'sentence': token.sent.text.strip() } # Traverse dependency tree for related tokens for child in token.children: # Check for drug as instrument (prepositional: "with amiodarone") if child.dep_ == "prep": for grandchild in child.children: if grandchild.lemma_.lower() in CARDIAC_DRUGS: relation['drug'] = grandchild.text # Check for patient as subject ("patient was extubated") if child.dep_ in ("nsubj", "nsubjpass"): if child.lemma_.lower() in ("patient",): relation['patient_subject'] = True # Check for outcome adverbs ("successfully cardioverted") if child.dep_ == "advmod": relation['outcome_modifier'] = child.text # Also check the token's own head for drugs for sibling in token.head.children: if sibling.lemma_.lower() in CARDIAC_DRUGS and sibling != token: if not relation['drug']: relation['drug'] = sibling.text if relation['verb']: relations.append(relation) return relations # Test on the postoperative section postop_text = """ The patient was extubated on POD 1 without complications. Chest tubes were removed on POD 2. Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone loading followed by maintenance dose. """ relations = extract_drug_event_relations(postop_text, nlp_sci) print("Drug-Event Relationships Extracted:") for r in relations: print(f"\\n Verb:{r['verb']}") print(f" Drug:{r['drug'] or 'N/A'}") print(f" Patient subject:{r['patient_subject']}") print(f" Outcome modifier:{r['outcome_modifier'] or 'N/A'}") print(f" Sentence:{r['sentence'][:80]}...") ``` The dependency parser thus enables extraction of _structured relationships_: not just entity mentions, but how entities interact within clinical events. This capability has direct applications in pharmacovigilance, complication tracking, and automated population of surgical registries. --- ## 3. scispaCy: Biomedical NLP ### 3.1 Why Standard Models Fail for Medical Text To appreciate scispaCy’s contribution, it helps to compare how a general-purpose spaCy model and a scispaCy model handle the same clinical sentence. Take: _“LIMA to LAD anastomosis was completed with satisfactory flow measurements.”_ **General spaCy (en\_core\_web\_sm):** - “LIMA” → labeled as `ORG` (organization), incorrect - “LAD” → labeled as `ORG`, incorrect - “anastomosis” → not recognized as a named entity at all **scispaCy (en\_core\_sci\_md):** - “LIMA” → recognized with `ENTITY` label (biomedical entity) - “LAD” → recognized as biomedical entity - “anastomosis” → recognized as biomedical entity The difference is training data. scispaCy models are trained on the CRAFT corpus (Colorado Richly Annotated Full-Text Corpus) and PubMed abstracts, scientific biomedical text that includes the vocabulary and entity patterns of clinical medicine. The NER annotations in these corpora rely on the BC5CDR schema for chemical and disease entities, plus broader entity annotations from ontologies like UMLS. ### 3.2 Named Entity Recognition with scispaCy Named Entity Recognition (NER) is the automatic identification and classification of named entities in text; in our case, medical concepts such as diseases, drugs, procedures, and anatomical structures. import spacy from typing import List, Dict, Tuple import json # Load scispaCy model nlp_sci = spacy.load("en_core_sci_md") # Full discharge summary for NER analysis DISCHARGE_SUMMARY = """ CARDIAC SURGERY DISCHARGE SUMMARY Patient: John Smith MRN: 12345678 DOB: 15/03/1958 Admission Date: 10/01/2024 Discharge Date: 18/01/2024 DIAGNOSIS: 1. Triple vessel coronary artery disease 2. Reduced left ventricular function (LVEF 35%) 3. Hypertension 4. Type 2 Diabetes Mellitus PROCEDURE PERFORMED: Coronary Artery Bypass Grafting x3 (CABG) - LIMA to LAD - SVG to OM1 - SVG to RCA Date of surgery: 11/01/2024 OPERATIVE DETAILS: The procedure was performed via median sternotomy under general anesthesia. Cardiopulmonary bypass time: 98 minutes Aortic cross-clamp time: 67 minutes All anastomoses were completed with satisfactory flow measurements. POSTOPERATIVE COURSE: The patient was extubated on POD 1 without complications. Chest tubes were removed on POD 2. Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone loading followed by maintenance dose. Patient remained in sinus rhythm thereafter. No wound infections or sternal instability noted. Echocardiography on POD 5 showed LVEF 40% with good biventricular function and no pericardial effusion. DISCHARGE MEDICATIONS: 1. Aspirin 100mg daily 2. Clopidogrel 75mg daily (for 12 months) 3. Atorvastatin 80mg daily 4. Metoprolol 50mg twice daily 5. Ramipril 5mg daily 6. Amiodarone 200mg daily (for 6 weeks) 7. Metformin 1000mg twice daily 8. Furosemide 40mg daily (for 2 weeks) FOLLOW-UP: - Outpatient cardiology clinic in 2 weeks - Cardiac rehabilitation program enrollment - INR monitoring not required (patient on dual antiplatelet therapy) The patient was discharged home in stable condition with appropriate wound care instructions. Dr. Sarah Johnson, MD Cardiac Surgery Department """ def extract_biomedical_entities(text: str, nlp) -> Dict[str, List[str]]: """ Extract all biomedical entities from clinical text using scispaCy NER. scispaCy's NER model identifies spans of text corresponding to biomedical concepts: diseases, chemicals, genes, cell types, species, and more. The entity label set depends on the model used: - en_core_sci_*: Uses a single 'ENTITY' label for all biomedical concepts - en_ner_bc5cdr_md: Distinguishes DISEASE and CHEMICAL entities - en_ner_jnlpba_md: Tags GENE, PROTEIN, RNA, DNA, CELL_TYPE, CELL_LINE For general clinical use, en_core_sci_md with UMLS linking provides the most complete semantic annotation. Args: text: Clinical text nlp: Loaded scispaCy model Returns: Dictionary grouping entities by their labels """ doc = nlp(text) entities_by_label: Dict[str, List[str]] = {} for ent in doc.ents: label = ent.label_ if label not in entities_by_label: entities_by_label[label] = [] # Deduplicate while preserving order entity_text = ent.text.strip() if entity_text and entity_text not in entities_by_label[label]: entities_by_label[label].append(entity_text) return entities_by_label # Extract entities from the discharge summary entities = extract_biomedical_entities(DISCHARGE_SUMMARY, nlp_sci) print("Biomedical Entities Extracted by scispaCy:") print("=" * 60) for label, ents in sorted(entities.items()): print(f"\\n[{label}] ({len(ents)} entities):") for entity in ents[:15]: # Show first 15 per category print(f" -{entity}") ``` import spacy from typing import List, Dict, Tuple import json # Load scispaCy model nlp_sci = spacy.load("en_core_sci_md") # Full discharge summary for NER analysis DISCHARGE_SUMMARY = """ CARDIAC SURGERY DISCHARGE SUMMARY Patient: John Smith MRN: 12345678 DOB: 15/03/1958 Admission Date: 10/01/2024 Discharge Date: 18/01/2024 DIAGNOSIS: 1. Triple vessel coronary artery disease 2. Reduced left ventricular function (LVEF 35%) 3. Hypertension 4. Type 2 Diabetes Mellitus PROCEDURE PERFORMED: Coronary Artery Bypass Grafting x3 (CABG) - LIMA to LAD - SVG to OM1 - SVG to RCA Date of surgery: 11/01/2024 OPERATIVE DETAILS: The procedure was performed via median sternotomy under general anesthesia. Cardiopulmonary bypass time: 98 minutes Aortic cross-clamp time: 67 minutes All anastomoses were completed with satisfactory flow measurements. POSTOPERATIVE COURSE: The patient was extubated on POD 1 without complications. Chest tubes were removed on POD 2. Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone loading followed by maintenance dose. Patient remained in sinus rhythm thereafter. No wound infections or sternal instability noted. Echocardiography on POD 5 showed LVEF 40% with good biventricular function and no pericardial effusion. DISCHARGE MEDICATIONS: 1. Aspirin 100mg daily 2. Clopidogrel 75mg daily (for 12 months) 3. Atorvastatin 80mg daily 4. Metoprolol 50mg twice daily 5. Ramipril 5mg daily 6. Amiodarone 200mg daily (for 6 weeks) 7. Metformin 1000mg twice daily 8. Furosemide 40mg daily (for 2 weeks) FOLLOW-UP: - Outpatient cardiology clinic in 2 weeks - Cardiac rehabilitation program enrollment - INR monitoring not required (patient on dual antiplatelet therapy) The patient was discharged home in stable condition with appropriate wound care instructions. Dr. Sarah Johnson, MD Cardiac Surgery Department """ def extract_biomedical_entities(text: str, nlp) -> Dict[str, List[str]]: """ Extract all biomedical entities from clinical text using scispaCy NER. scispaCy's NER model identifies spans of text corresponding to biomedical concepts: diseases, chemicals, genes, cell types, species, and more. The entity label set depends on the model used: - en_core_sci_*: Uses a single 'ENTITY' label for all biomedical concepts - en_ner_bc5cdr_md: Distinguishes DISEASE and CHEMICAL entities - en_ner_jnlpba_md: Tags GENE, PROTEIN, RNA, DNA, CELL_TYPE, CELL_LINE For general clinical use, en_core_sci_md with UMLS linking provides the most complete semantic annotation. Args: text: Clinical text nlp: Loaded scispaCy model Returns: Dictionary grouping entities by their labels """ doc = nlp(text) entities_by_label: Dict[str, List[str]] = {} for ent in doc.ents: label = ent.label_ if label not in entities_by_label: entities_by_label[label] = [] # Deduplicate while preserving order entity_text = ent.text.strip() if entity_text and entity_text not in entities_by_label[label]: entities_by_label[label].append(entity_text) return entities_by_label # Extract entities from the discharge summary entities = extract_biomedical_entities(DISCHARGE_SUMMARY, nlp_sci) print("Biomedical Entities Extracted by scispaCy:") print("=" * 60) for label, ents in sorted(entities.items()): print(f"\\n[{label}] ({len(ents)} entities):") for entity in ents[:15]: # Show first 15 per category print(f" -{entity}") ``` ### 3.3 Specialized NER Models: BC5CDR for Diseases and Chemicals For clinical NLP requiring a precise distinction between disease entities and chemical/drug entities, the `en_ner_bc5cdr_md` scispaCy model trained on the BioCreative V CDR corpus is the appropriate choice. The model was trained to distinguish: - **DISEASE**: Pathological conditions (“atrial fibrillation,” “coronary artery disease,” “hypertension”) - **CHEMICAL**: Drugs and chemical compounds (“amiodarone,” “aspirin,” “atorvastatin”) # Note: requires installation of en_ner_bc5cdr_md # pip install def extract_diseases_and_chemicals(text: str) -> Dict[str, List[str]]: """ Use the BC5CDR model to separately identify diseases and drugs. The BC5CDR model was trained on 1,500 PubMed articles with expert annotations of disease and chemical mentions. It achieves F1 scores >85% on the BC5CDR test set, making it suitable for clinical applications where chemical-disease distinction matters. Clinical application: Automatically extracting the complication profile and pharmacological treatment from discharge summaries for pharmacovigilance databases. Args: text: Clinical document text Returns: Dictionary with 'DISEASE' and 'CHEMICAL' entity lists """ try: nlp_cdr = spacy.load("en_ner_bc5cdr_md") except OSError: print("Model en_ner_bc5cdr_md not installed. Using en_core_sci_md instead.") return extract_biomedical_entities(text, spacy.load("en_core_sci_md")) doc = nlp_cdr(text) result = {"DISEASE": [], "CHEMICAL": []} for ent in doc.ents: entity_text = ent.text.strip().lower() label = ent.label_ if label in result and entity_text not in result[label]: result[label].append(ent.text.strip()) return result # Test on our discharge summary entities_bc5cdr = extract_diseases_and_chemicals(DISCHARGE_SUMMARY) print("Diseases detected:") for disease in entities_bc5cdr.get("DISEASE", []): print(f" ✓{disease}") print("\\nChemicals/Drugs detected:") for chemical in entities_bc5cdr.get("CHEMICAL", []): print(f" ✓{chemical}") ``` # Note: requires installation of en_ner_bc5cdr_md # pip install def extract_diseases_and_chemicals(text: str) -> Dict[str, List[str]]: """ Use the BC5CDR model to separately identify diseases and drugs. The BC5CDR model was trained on 1,500 PubMed articles with expert annotations of disease and chemical mentions. It achieves F1 scores >85% on the BC5CDR test set, making it suitable for clinical applications where chemical-disease distinction matters. Clinical application: Automatically extracting the complication profile and pharmacological treatment from discharge summaries for pharmacovigilance databases. Args: text: Clinical document text Returns: Dictionary with 'DISEASE' and 'CHEMICAL' entity lists """ try: nlp_cdr = spacy.load("en_ner_bc5cdr_md") except OSError: print("Model en_ner_bc5cdr_md not installed. Using en_core_sci_md instead.") return extract_biomedical_entities(text, spacy.load("en_core_sci_md")) doc = nlp_cdr(text) result = {"DISEASE": [], "CHEMICAL": []} for ent in doc.ents: entity_text = ent.text.strip().lower() label = ent.label_ if label in result and entity_text not in result[label]: result[label].append(ent.text.strip()) return result # Test on our discharge summary entities_bc5cdr = extract_diseases_and_chemicals(DISCHARGE_SUMMARY) print("Diseases detected:") for disease in entities_bc5cdr.get("DISEASE", []): print(f" ✓{disease}") print("\\nChemicals/Drugs detected:") for chemical in entities_bc5cdr.get("CHEMICAL", []): print(f" ✓{chemical}") ``` ### 3.4 Abbreviation Detection and Expansion Medical text is dense with abbreviations, a major source of ambiguity for NLP systems. “AF” in a cardiology report almost certainly means atrial fibrillation; in a different context, it might mean acid-fast, amniotic fluid, or audio frequency. scispaCy provides an `AbbreviationDetector` component that identifies abbreviation-definition pairs within the same document, enabling in-context disambiguation. import spacy import scispacy from scispacy.abbreviation import AbbreviationDetector def setup_abbreviation_detector(model_name: str = "en_core_sci_md") -> spacy.language.Language: """ Configure scispaCy pipeline with abbreviation detection. The AbbreviationDetector implements the algorithm from Schwartz & Hearst (2003), which identifies abbreviation-definition pairs based on character matching heuristics. When a document defines "CABG" as "Coronary Artery Bypass Grafting", the detector enables automatic expansion of all subsequent CABG mentions. Returns: spaCy language model with abbreviation detection configured """ nlp = spacy.load(model_name) # Add abbreviation detector to the pipeline nlp.add_pipe("abbreviation_detector") return nlp def extract_abbreviations(text: str, nlp) -> Dict[str, str]: """ Extract and map abbreviations found in clinical text. Medical abbreviations are a significant challenge for NLP systems: the same abbreviation can have different meanings in different specialties or even different sections of the same document. This function identifies abbreviation-definition pairs where both the abbreviation and its expansion appear in the same document. Args: text: Clinical document text nlp: scispaCy model with AbbreviationDetector Returns: Dictionary mapping abbreviation → full form """ doc = nlp(text) abbreviation_map = {} for abbreviation in doc._.abbreviations: abbr_text = abbreviation.text # The long form is the identified full expansion long_form = abbreviation._.long_form.text if abbreviation._.long_form else "Unknown" abbreviation_map[abbr_text] = long_form return abbreviation_map # Note: Our discharge summary uses abbreviations but may not always define them inline # Let's create a version with explicit definitions to demonstrate the capability annotated_summary = """ The patient underwent Coronary Artery Bypass Grafting (CABG) using Left Internal Mammary Artery (LIMA) to Left Anterior Descending (LAD) and Saphenous Vein Graft (SVG) to Obtuse Marginal (OM1) and Right Coronary Artery (RCA). Cardiopulmonary Bypass (CPB) time was 98 minutes. Postoperative atrial fibrillation (AF) was noted and treated with amiodarone. Left Ventricular Ejection Fraction (LVEF) improved from 35% to 40%. """ nlp_with_abbrev = setup_abbreviation_detector() abbreviations = extract_abbreviations(annotated_summary, nlp_with_abbrev) print("Abbreviations detected and expanded:") for abbr, expansion in sorted(abbreviations.items()): print(f"{abbr} →{expansion}") ``` import spacy import scispacy from scispacy.abbreviation import AbbreviationDetector def setup_abbreviation_detector(model_name: str = "en_core_sci_md") -> spacy.language.Language: """ Configure scispaCy pipeline with abbreviation detection. The AbbreviationDetector implements the algorithm from Schwartz & Hearst (2003), which identifies abbreviation-definition pairs based on character matching heuristics. When a document defines "CABG" as "Coronary Artery Bypass Grafting", the detector enables automatic expansion of all subsequent CABG mentions. Returns: spaCy language model with abbreviation detection configured """ nlp = spacy.load(model_name) # Add abbreviation detector to the pipeline nlp.add_pipe("abbreviation_detector") return nlp def extract_abbreviations(text: str, nlp) -> Dict[str, str]: """ Extract and map abbreviations found in clinical text. Medical abbreviations are a significant challenge for NLP systems: the same abbreviation can have different meanings in different specialties or even different sections of the same document. This function identifies abbreviation-definition pairs where both the abbreviation and its expansion appear in the same document. Args: text: Clinical document text nlp: scispaCy model with AbbreviationDetector Returns: Dictionary mapping abbreviation → full form """ doc = nlp(text) abbreviation_map = {} for abbreviation in doc._.abbreviations: abbr_text = abbreviation.text # The long form is the identified full expansion long_form = abbreviation._.long_form.text if abbreviation._.long_form else "Unknown" abbreviation_map[abbr_text] = long_form return abbreviation_map # Note: Our discharge summary uses abbreviations but may not always define them inline # Let's create a version with explicit definitions to demonstrate the capability annotated_summary = """ The patient underwent Coronary Artery Bypass Grafting (CABG) using Left Internal Mammary Artery (LIMA) to Left Anterior Descending (LAD) and Saphenous Vein Graft (SVG) to Obtuse Marginal (OM1) and Right Coronary Artery (RCA). Cardiopulmonary Bypass (CPB) time was 98 minutes. Postoperative atrial fibrillation (AF) was noted and treated with amiodarone. Left Ventricular Ejection Fraction (LVEF) improved from 35% to 40%. """ nlp_with_abbrev = setup_abbreviation_detector() abbreviations = extract_abbreviations(annotated_summary, nlp_with_abbrev) print("Abbreviations detected and expanded:") for abbr, expansion in sorted(abbreviations.items()): print(f"{abbr} →{expansion}") ``` **Expected output:** ``` Abbreviations detected and expanded: AF → atrial fibrillation CABG → Coronary Artery Bypass Grafting CPB → Cardiopulmonary Bypass LAD → Left Anterior Descending LIMA → Left Internal Mammary Artery LVEF → Left Ventricular Ejection Fraction OM1 → Obtuse Marginal RCA → Right Coronary Artery SVG → Saphenous Vein Graft ``` A caveat from clinical practice: the Schwartz & Hearst algorithm only catches abbreviations that are _explicitly defined inside the same document_. Most operative reports we read in our daily work assume the reader already knows what CABG, LIMA or LVEF mean and never spell them out. For real deployment, this detector almost always needs to be paired with a curated cardiac-surgery glossary as a fallback. --- ## 4. UMLS Entity Linking: Connecting to Medical Ontologies ### 4.1 The Importance of Standardized Medical Vocabulary Named entity recognition tells us _that_ a text span refers to a medical concept; UMLS linking tells us _which_ standardized concept it refers to. The distinction matters for interoperability. Consider: “coronary artery disease,” “CAD,” “ischemic heart disease,” and “arteriosclerotic heart disease” are all strings pointing to the same underlying condition. In the Unified Medical Language System (UMLS), this concept has a single Concept Unique Identifier: **C0010068**. By linking extracted entities to UMLS CUIs we get: 1. **Semantic normalization**: All equivalent surface forms map to the same concept 2. **Cross-database interoperability**: UMLS bridges SNOMED-CT, ICD-10, MeSH, RxNorm, and thousands of other vocabularies 3. **Knowledge graph integration**: CUIs can be used to query external medical knowledge bases 4. **Research reproducibility**: Studies using CUI-based cohort definitions are unambiguous The UMLS Metathesaurus is maintained by the U.S. National Library of Medicine and covers over 3.5 million concepts across more than 200 biomedical vocabularies. Key semantic type hierarchies relevant to cardiac surgery include: - **T047**, Disease or Syndrome (coronary artery disease, atrial fibrillation) - **T121**, Pharmacologic Substance (amiodarone, aspirin) - **T061**, Therapeutic or Preventive Procedure (CABG, cardioversion) - **T023**, Body Part, Organ, or Organ Component (left ventricle, coronary artery) ### 4.2 Implementing UMLS Entity Linking with scispaCy import spacy import scispacy from scispacy.linking import EntityLinker def setup_umls_pipeline(model_name: str = "en_core_sci_md") -> spacy.language.Language: """ Configure scispaCy pipeline with UMLS entity linking. The EntityLinker component compares extracted entity spans against a local copy of the UMLS knowledge base using approximate string matching and concept embeddings. It returns ranked candidate CUIs with confidence scores. IMPORTANT: The UMLS linker requires ~2GB RAM for the knowledge base and has a one-time download of ~1.5GB. Performance is substantially better with en_core_sci_lg than with the smaller models. Configuration parameters: - resolve_abbreviations: Use AbbreviationDetector before linking - linker_name: "umls" uses full UMLS; "mesh" uses MeSH only (smaller) - threshold: Minimum similarity score (0.7-0.85 recommended for clinical use) - max_entities_per_mention: Candidates per entity span Returns: Configured spaCy pipeline with UMLS linking """ nlp = spacy.load(model_name) # Add abbreviation detection (must come before linker) nlp.add_pipe("abbreviation_detector") # Add UMLS entity linker nlp.add_pipe( "scispacy_linker", config={ "resolve_abbreviations": True, # Expand abbreviations before linking "linker_name": "umls", # Full UMLS knowledge base "threshold": 0.80, # Minimum similarity threshold "max_entities_per_mention": 3, # Top-k candidates } ) return nlp def extract_umls_entities(text: str, nlp) -> List[Dict]: """ Extract entities with full UMLS annotations. For each recognized biomedical entity, this function retrieves: - The entity text as it appears in the document - The UMLS Concept Unique Identifier (CUI) - The canonical (preferred) name for the concept - The UMLS semantic type (T047 = Disease, T121 = Drug, etc.) - The similarity score (confidence of the linking) - The definition from UMLS (if available) Args: text: Clinical document text nlp: Pipeline with UMLS linker configured Returns: List of entities with complete UMLS annotations """ doc = nlp(text) # Access the linker component for knowledge base queries linker = nlp.get_pipe("scispacy_linker") annotated_entities = [] for ent in doc.ents: entity_info = { 'text': ent.text, 'start_char': ent.start_char, 'end_char': ent.end_char, 'label': ent.label_, 'umls_candidates': [] } # Get UMLS linking results for umls_entity in ent._.kb_ents: cui = umls_entity[0] # UMLS Concept Unique Identifier score = umls_entity[1] # Similarity score (0-1) # Query the knowledge base for this CUI if cui in linker.kb.cui_to_entity: kb_entity = linker.kb.cui_to_entity[cui] candidate = { 'cui': cui, 'canonical_name': kb_entity.canonical_name, 'aliases': list(kb_entity.aliases[:5]), # First 5 aliases 'types': list(kb_entity.types), # Semantic type codes 'definition': kb_entity.definition or "No definition available", 'similarity_score': round(score, 4) } entity_info['umls_candidates'].append(candidate) if entity_info['umls_candidates']: annotated_entities.append(entity_info) return annotated_entities def format_umls_report(entities: List[Dict]) -> str: """ Format UMLS entity linking results as a readable clinical report. Args: entities: Output from extract_umls_entities() Returns: Formatted string report """ lines = ["UMLS Entity Linking Report", "=" * 60] # Semantic type descriptions for readability SEMANTIC_TYPES = { "T047": "Disease or Syndrome", "T121": "Pharmacologic Substance", "T061": "Therapeutic/Preventive Procedure", "T023": "Body Part, Organ, Component", "T116": "Amino Acid/Peptide/Protein", "T048": "Mental/Behavioral Dysfunction", "T033": "Finding", "T184": "Sign or Symptom", "T060": "Diagnostic Procedure", } for entity in entities: lines.append(f"\\nEntity: '{entity['text']}'") lines.append(f" Character span: [{entity['start_char']}:{entity['end_char']}]") if entity['umls_candidates']: best = entity['umls_candidates'][0] # Top candidate lines.append(f" Best UMLS match:") lines.append(f" CUI:{best['cui']}") lines.append(f" Canonical name:{best['canonical_name']}") lines.append(f" Similarity:{best['similarity_score']:.2%}") # Describe semantic types type_descriptions = [ SEMANTIC_TYPES.get(t, t) for t in best['types'] ] lines.append(f" Semantic types:{', '.join(type_descriptions)}") if best['aliases']: lines.append(f" Known aliases:{', '.join(best['aliases'][:3])}") return "\\n".join(lines) # Example usage # Note: requires ~2GB RAM for UMLS knowledge base # nlp_umls = setup_umls_pipeline() # entities = extract_umls_entities(DISCHARGE_SUMMARY, nlp_umls) # print(format_umls_report(entities)) ``` import spacy import scispacy from scispacy.linking import EntityLinker def setup_umls_pipeline(model_name: str = "en_core_sci_md") -> spacy.language.Language: """ Configure scispaCy pipeline with UMLS entity linking. The EntityLinker component compares extracted entity spans against a local copy of the UMLS knowledge base using approximate string matching and concept embeddings. It returns ranked candidate CUIs with confidence scores. IMPORTANT: The UMLS linker requires ~2GB RAM for the knowledge base and has a one-time download of ~1.5GB. Performance is substantially better with en_core_sci_lg than with the smaller models. Configuration parameters: - resolve_abbreviations: Use AbbreviationDetector before linking - linker_name: "umls" uses full UMLS; "mesh" uses MeSH only (smaller) - threshold: Minimum similarity score (0.7-0.85 recommended for clinical use) - max_entities_per_mention: Candidates per entity span Returns: Configured spaCy pipeline with UMLS linking """ nlp = spacy.load(model_name) # Add abbreviation detection (must come before linker) nlp.add_pipe("abbreviation_detector") # Add UMLS entity linker nlp.add_pipe( "scispacy_linker", config={ "resolve_abbreviations": True, # Expand abbreviations before linking "linker_name": "umls", # Full UMLS knowledge base "threshold": 0.80, # Minimum similarity threshold "max_entities_per_mention": 3, # Top-k candidates } ) return nlp def extract_umls_entities(text: str, nlp) -> List[Dict]: """ Extract entities with full UMLS annotations. For each recognized biomedical entity, this function retrieves: - The entity text as it appears in the document - The UMLS Concept Unique Identifier (CUI) - The canonical (preferred) name for the concept - The UMLS semantic type (T047 = Disease, T121 = Drug, etc.) - The similarity score (confidence of the linking) - The definition from UMLS (if available) Args: text: Clinical document text nlp: Pipeline with UMLS linker configured Returns: List of entities with complete UMLS annotations """ doc = nlp(text) # Access the linker component for knowledge base queries linker = nlp.get_pipe("scispacy_linker") annotated_entities = [] for ent in doc.ents: entity_info = { 'text': ent.text, 'start_char': ent.start_char, 'end_char': ent.end_char, 'label': ent.label_, 'umls_candidates': [] } # Get UMLS linking results for umls_entity in ent._.kb_ents: cui = umls_entity[0] # UMLS Concept Unique Identifier score = umls_entity[1] # Similarity score (0-1) # Query the knowledge base for this CUI if cui in linker.kb.cui_to_entity: kb_entity = linker.kb.cui_to_entity[cui] candidate = { 'cui': cui, 'canonical_name': kb_entity.canonical_name, 'aliases': list(kb_entity.aliases[:5]), # First 5 aliases 'types': list(kb_entity.types), # Semantic type codes 'definition': kb_entity.definition or "No definition available", 'similarity_score': round(score, 4) } entity_info['umls_candidates'].append(candidate) if entity_info['umls_candidates']: annotated_entities.append(entity_info) return annotated_entities def format_umls_report(entities: List[Dict]) -> str: """ Format UMLS entity linking results as a readable clinical report. Args: entities: Output from extract_umls_entities() Returns: Formatted string report """ lines = ["UMLS Entity Linking Report", "=" * 60] # Semantic type descriptions for readability SEMANTIC_TYPES = { "T047": "Disease or Syndrome", "T121": "Pharmacologic Substance", "T061": "Therapeutic/Preventive Procedure", "T023": "Body Part, Organ, Component", "T116": "Amino Acid/Peptide/Protein", "T048": "Mental/Behavioral Dysfunction", "T033": "Finding", "T184": "Sign or Symptom", "T060": "Diagnostic Procedure", } for entity in entities: lines.append(f"\\nEntity: '{entity['text']}'") lines.append(f" Character span: [{entity['start_char']}:{entity['end_char']}]") if entity['umls_candidates']: best = entity['umls_candidates'][0] # Top candidate lines.append(f" Best UMLS match:") lines.append(f" CUI:{best['cui']}") lines.append(f" Canonical name:{best['canonical_name']}") lines.append(f" Similarity:{best['similarity_score']:.2%}") # Describe semantic types type_descriptions = [ SEMANTIC_TYPES.get(t, t) for t in best['types'] ] lines.append(f" Semantic types:{', '.join(type_descriptions)}") if best['aliases']: lines.append(f" Known aliases:{', '.join(best['aliases'][:3])}") return "\\n".join(lines) # Example usage # Note: requires ~2GB RAM for UMLS knowledge base # nlp_umls = setup_umls_pipeline() # entities = extract_umls_entities(DISCHARGE_SUMMARY, nlp_umls) # print(format_umls_report(entities)) ``` **Expected output for key entities** (illustrative, actual CUIs from UMLS 2024AB): UMLS Entity Linking Report Entity: ‘coronary artery disease’ Character span: \[150:172\] Best UMLS match: CUI: C0010068 Canonical name: Coronary Arteriosclerosis Similarity: 96.80% Semantic types: Disease or Syndrome Known aliases: CAD, Ischemic Heart Disease, Coronary Artery Disease Entity: ‘atrial fibrillation’ Character span: \[478:496\] Best UMLS match: CUI: C0004238 Canonical name: Atrial Fibrillation Similarity: 99.10% Semantic types: Disease or Syndrome, Finding Known aliases: AF, A-fib, Auricular Fibrillation Entity: ‘amiodarone’ Character span: \[521:531\] Best UMLS match: CUI: C0002598 Canonical name: Amiodarone Similarity: 99.90% Semantic types: Pharmacologic Substance Known aliases: Cordarone, Nexterone, Pacerone Entity: ‘median sternotomy’ Best UMLS match: CUI: C0185792 Canonical name: Median Sternotomy Similarity: 98.50% Semantic types: Therapeutic/Preventive Procedure --- ## 5. Temporal Information Extraction ### 5.1 The Clinical Importance of Temporal Relations In cardiac surgery, the timing of events carries weight comparable to the events themselves. “Atrial fibrillation on POD 3” and “atrial fibrillation on POD 14” may imply different etiologies, different risks, and different management approaches. A complete NLP pipeline therefore needs to capture not only _what_ happened, but _when_. spaCy’s dependency parser is well suited to extracting temporal relationships: def extract_temporal_clinical_events(text: str, nlp) -> List[Dict]: """ Extract clinical events with their temporal anchors. Uses dependency parsing to identify relationships between clinical events and their associated postoperative day (POD), time expressions, or date references. Temporal patterns in cardiac surgery discharge summaries: - Explicit POD references: "extubated on POD 1" - Implicit timing: "chest tubes removed subsequently" - Duration: "amiodarone for 6 weeks" - Relative timing: "48 hours after surgery" Args: text: Clinical narrative text nlp: scispaCy model Returns: List of events with temporal information """ doc = nlp(text) # Pattern for "POD N", postoperative day pod_pattern = re.compile(r'POD\\s*(\\d+)', re.IGNORECASE) events_with_timing = [] for sent in doc.sents: sent_text = sent.text # Find POD references in this sentence pod_matches = pod_pattern.findall(sent_text) pod_days = [int(d) for d in pod_matches] # Find clinical events (non-stop verbs and nouns) clinical_tokens = [] for token in sent: if (token.pos_ in ("VERB", "NOUN", "PROPN") and not token.is_stop and len(token.text) > 2): clinical_tokens.append(token.lemma_) if pod_days and clinical_tokens: events_with_timing.append({ 'sentence': sent_text.strip(), 'pod_days': pod_days, 'earliest_pod': min(pod_days), 'clinical_tokens': clinical_tokens[:8], # Top 8 content words }) # Sort by earliest POD events_with_timing.sort(key=lambda x: x['earliest_pod']) return events_with_timing # Build clinical timeline from the postoperative course postop_text = """ The patient was extubated on POD 1 without complications. Chest tubes were removed on POD 2. Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone loading followed by maintenance dose. Patient remained in sinus rhythm thereafter. Echocardiography on POD 5 showed LVEF 40% with good biventricular function. """ timeline = extract_temporal_clinical_events(postop_text, nlp_sci) print("Clinical Timeline Reconstruction:") print("=" * 60) for event in timeline: print(f"\\n POD{event['earliest_pod']}:") print(f"{event['sentence']}") ``` def extract_temporal_clinical_events(text: str, nlp) -> List[Dict]: """ Extract clinical events with their temporal anchors. Uses dependency parsing to identify relationships between clinical events and their associated postoperative day (POD), time expressions, or date references. Temporal patterns in cardiac surgery discharge summaries: - Explicit POD references: "extubated on POD 1" - Implicit timing: "chest tubes removed subsequently" - Duration: "amiodarone for 6 weeks" - Relative timing: "48 hours after surgery" Args: text: Clinical narrative text nlp: scispaCy model Returns: List of events with temporal information """ doc = nlp(text) # Pattern for "POD N", postoperative day pod_pattern = re.compile(r'POD\\s*(\\d+)', re.IGNORECASE) events_with_timing = [] for sent in doc.sents: sent_text = sent.text # Find POD references in this sentence pod_matches = pod_pattern.findall(sent_text) pod_days = [int(d) for d in pod_matches] # Find clinical events (non-stop verbs and nouns) clinical_tokens = [] for token in sent: if (token.pos_ in ("VERB", "NOUN", "PROPN") and not token.is_stop and len(token.text) > 2): clinical_tokens.append(token.lemma_) if pod_days and clinical_tokens: events_with_timing.append({ 'sentence': sent_text.strip(), 'pod_days': pod_days, 'earliest_pod': min(pod_days), 'clinical_tokens': clinical_tokens[:8], # Top 8 content words }) # Sort by earliest POD events_with_timing.sort(key=lambda x: x['earliest_pod']) return events_with_timing # Build clinical timeline from the postoperative course postop_text = """ The patient was extubated on POD 1 without complications. Chest tubes were removed on POD 2. Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone loading followed by maintenance dose. Patient remained in sinus rhythm thereafter. Echocardiography on POD 5 showed LVEF 40% with good biventricular function. """ timeline = extract_temporal_clinical_events(postop_text, nlp_sci) print("Clinical Timeline Reconstruction:") print("=" * 60) for event in timeline: print(f"\\n POD{event['earliest_pod']}:") print(f"{event['sentence']}") ``` **Output:** ``` Clinical Timeline Reconstruction: ============================================================ POD 1: The patient was extubated on POD 1 without complications. POD 2: Chest tubes were removed on POD 2. POD 3: Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone loading. POD 5: Echocardiography on POD 5 showed LVEF 40% with good biventricular function. ``` This kind of structured temporal reconstruction has direct applications in automated quality metrics reporting, particularly for surgical safety indicators like time-to-extubation and time-to-complication onset. --- ## 6. Comparing spaCy/scispaCy with Regex: A Decision Framework ### 6.1 Quantitative Performance Considerations Neither regex nor linguistic NLP is universally better. The optimal choice depends on the specific extraction task, the consistency of the source documents, and the computational resources you actually have at your disposal. The table below summarizes the practical considerations: | Criterion | Regex | spaCy/scispaCy | | --- | --- | --- | | **Setup complexity** | Minimal | Moderate (model installation) | | **Processing speed** | Very fast (~1ms/doc) | Moderate (100ms–5s/doc) | | **Memory requirements** | Negligible | 500MB–2GB | | **Accuracy for structured fields** | High (when patterns are known) | Moderate | | **Accuracy for free-text narrative** | Low | High | | **Negation handling** | Manual, brittle | Built-in via dependency parsing | | **Semantic normalization** | None | UMLS linking | | **Maintenance burden** | High (pattern updates) | Low (model updates) | | **Explainability** | Complete | Partial (neural models) | | **Regulatory compliance** | Straightforward | Requires validation | ### 6.2 When to Use Regex Regex is still the preferred approach for: - **Highly structured fields** with predictable formats: dates, MRN numbers, medication dosages in formatted lists - **Real-time applications** where processing latency is critical - **Resource-constrained environments** (edge computing, embedded systems) - **Deterministic requirements** where output must be fully explainable and reproducible - **Simple extraction rules** with few variations (e.g., “extract all numbers followed by ‘mg’”) ### 6.3 When to Use spaCy/scispaCy Linguistic NLP earns its place when you need: - **Free-text narratives** with high syntactic variability (postoperative course sections) - **Entity recognition** where surface forms vary (“AF,” “atrial fib,” “atrial fibrillation”) - **Relationship extraction** between medical concepts (drug → indication, procedure → complication) - **Negation detection** at scale (well above what rule-based approaches can deliver) - **Interoperability requirements** where UMLS/SNOMED-CT standardization is needed - **Research applications** requiring semantic search or concept-based cohort selection ### 6.4 The Hybrid Pipeline (Preview) In practice, the most robust systems combine both. A hybrid pipeline for cardiac surgery discharge summaries might follow this logic: 1. **Regex layer**: Extract structured fields (dates, numeric values, section headers) with high precision 2. **scispaCy NER layer**: Identify medical entities in free-text sections 3. **UMLS linking layer**: Normalize entities to standard vocabularies 4. **Validation layer**: Cross-validate regex and NLP results; flag discrepancies The architecture is explored in depth in Article 3, where we layer in machine learning classifiers for complication severity scoring and LLM-based reasoning for the harder inference tasks. --- ## 7. Conclusion: From Syntax to Semantics This article has shown how spaCy and scispaCy push medical text analysis past character-level pattern matching, into something closer to genuine linguistic intelligence. The key advances over regex-based extraction are: **Semantic awareness**: scispaCy NER recognizes medical concepts regardless of their surface form, handling the terminological variability inherent to clinical language. “Triple vessel coronary artery disease,” “3-vessel CAD,” and “severe multivessel coronary disease” all refer to the same clinical condition. **Relational extraction**: Dependency parsing exposes the grammatical relationships between tokens, enabling extraction of drug-event associations, procedural outcomes, and temporal sequences that regex simply cannot reach. **Ontological grounding**: UMLS entity linking standardizes extracted concepts to universal identifiers, supporting interoperability across electronic health record systems, clinical registries, and research databases. Once a CUI is assigned to a concept, that annotation is unambiguous across institutions and across time. **Negation sensitivity**: Medical NLP that cannot distinguish affirmed from negated findings is, plainly, clinically dangerous. The dependency-based negation detection illustrated here, while not exhaustive, handles the majority of common patterns in discharge documentation. ### Limitations of the Linguistic Approach scispaCy, biomedical training notwithstanding, is not without limitations that need to be acknowledged before any clinical use: - **Out-of-vocabulary terms**: Novel drug names, rare procedures, or institution-specific abbreviations may not be recognized - **Domain shift**: Models trained on PubMed abstracts may underperform on clinical notes, which have different linguistic characteristics - **Negation complexity**: Advanced negation patterns (“unlikely to represent,” “cannot exclude”) require specialized components (MedSpaCy, NegEx) beyond standard dependency parsing - **Numerical relationship extraction**: With “LVEF 35%”, scispaCy recognizes LVEF as an entity but does not inherently link the numerical value; that gap is best filled by regex or a post-processing layer - **Validation requirements**: Clinical deployment of NLP systems requires rigorous validation on institution-specific documents, with performance metrics (precision, recall, F1) computed on expert-annotated gold standards ## 8. Looking Ahead: Machine Learning and LLMs The third and final article in this series will tackle what neither regex nor linguistic NLP can do: learn from data, handle unrestricted text variation, and perform complex clinical inference. Machine learning classifiers trained on annotated discharge summaries can pick up complication risk from subtle linguistic patterns. Large Language Models (LLMs), the Claude API among them, can extract arbitrarily complex information through natural language querying, compare findings against clinical guidelines, and generate structured summaries from narrative text. The article will also address the non-negotiable topics: GDPR/HIPAA compliance, de-identification pipelines, and deployment considerations for production medical AI systems. **Preview**: scispaCy might recognize “atrial fibrillation” as a disease entity. A properly prompted LLM, by contrast, can determine whether the documented management (amiodarone loading, cardioversion) was consistent with current ACC/AHA guidelines, a level of clinical reasoning that pattern-based NLP systems cannot reach. ## References 1. Neumann M, et al. “ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing.” _Proceedings of the 18th BioNLP Workshop and Shared Task_, 2019:319–327. 2. Honnibal M, Montani I. “spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.” _Unpublished_, 2017. 3. Bodenreider O. “The Unified Medical Language System (UMLS): integrating biomedical terminology.” _Nucleic Acids Research_ 2004;32(suppl\_1):D267–D270. 4. Schwartz AS, Hearst MA. “A simple algorithm for identifying abbreviation definitions in biomedical text.” _Proceedings of the Pacific Symposium on Biocomputing_, 2003:451–462. 5. Li J, et al. “BioCreative V CDR task corpus: a resource for chemical disease relation extraction.” _Database_ 2016:baw068. 6. Lample G, et al. “Neural Architectures for Named Entity Recognition.” _Proceedings of NAACL-HLT 2016_, 2016:260–270. 7. Chapman WW, et al. “A simple algorithm for identifying negated findings and diseases in discharge summaries.” _Journal of Biomedical Informatics_ 2001;34(5):301–310. \[NegEx algorithm\] 8. Soldaini L, Goharian N. “QuickUMLS: a fast, unsupervised approach for medical concept extraction.” _MedIR Workshop, SIGIR_, 2016. 9. Peng Y, et al. “Transfer Learning in Biomedical Natural Language Processing.” _Proceedings of the 18th BioNLP Workshop_, 2019. 10. Savova GK, et al. “Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications.” _JAMIA_ 2010;17(5):507–513. 11. Uzuner O, et al. “2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text.” _JAMIA_ 2011;18(5):552–556. 12. Stenetorp P, et al. “BRAT: a Web-based Tool for NLP-Assisted Text Annotation.” _Proceedings of EACL 2012_, 2012:102–107.