Introduction to Natural Language Processing for Medical Report Analysis: Advanced Linguistic Analysis with spaCy and scispaCy for Medical Reports

Article authored by Michele D. Pierri, MD

Cardiac Surgeon & Medical Technology Researcher

Last updated: May 2025

Reading time: 30 minutes

Abstract

Regular expressions give us a solid foundation for pulling structured data out of clinical documents. They struggle, though, with the semantic complexity that defines medical language. This article introduces two complementary libraries (spaCy and scispaCy) that bring genuine linguistic intelligence to clinical text. We work through detailed examples on a post-CABG discharge summary, covering tokenization, part-of-speech tagging, dependency parsing, and Named Entity Recognition (NER) with biomedical models. UMLS entity linking is also explored, since it ties extracted clinical terms to standardized medical ontologies and enables interoperability across health information systems. By the final section the reader will know not only how to implement these techniques in Python, but when to prefer them over regex; and when, in the daily reality of clinical NLP, the two should be combined.

Article Series:

Introduction to NLP and Regex for Medical Reports
Advanced Linguistic Analysis with spaCy and scispaCy
Machine Learning and Large Language Models for Clinical Text Analysis

1. Beyond Pattern Matching: Why Linguistic Analysis?

1.1 The Limits of Regular Expressions

The first article of this series built a regex parser able to extract patient demographics, surgical parameters, medications, and complications from a post-CABG discharge summary. That implementation worked because our sample document followed predictable formatting conventions. Real-world clinical documentation, in our experience reading hundreds of operative reports across centers, is far less disciplined.

Consider these semantically equivalent phrases that all describe the same complication:

"Postoperative atrial fibrillation was noted on POD 3"
"Patient developed new-onset AF in the immediate post-surgical period"
"Rhythm monitoring revealed paroxysmal atrial fibrillation 72 hours after bypass"
"POD 3: AF, cardioverted successfully"

"Postoperative atrial fibrillation was noted on POD 3"
"Patient developed new-onset AF in the immediate post-surgical period"
"Rhythm monitoring revealed paroxysmal atrial fibrillation 72 hours after bypass"
"POD 3: AF, cardioverted successfully"

A regex pattern designed to catch the first phrasing will miss the others. We could write extra patterns for each variant, but this becomes an arms race against the variability of natural language. Regex also cannot:

Determine that “no wound infections” and “wound infection noted” have opposite clinical meanings (negation handling)
Understand that “CABG,” “bypass surgery,” and “coronary revascularization” refer to the same procedure
Extract the relationship between a drug and its indication (“amiodarone for atrial fibrillation”)
Handle abbreviation disambiguation (“AF” could mean atrial fibrillation or aortic flow depending on context)

These limitations are what push us toward NLP tools built on computational linguistics, the scientific study of language structure.

1.2 The Linguistic Approach

Natural language processing at the linguistic level rests on a simple premise: language has internal structure, and that structure can be computationally modeled. Rather than matching character sequences, linguistic NLP:

Tokenizes text into meaningful units (words, punctuation, medical abbreviations)
Tags each token with its grammatical role (noun, verb, adjective)
Parses the syntactic relationships between tokens (subject, object, modifier)
Recognizes named entities, that is, spans of text referring to real-world concepts (diseases, drugs, procedures)
Links recognized entities to external knowledge bases (UMLS, SNOMED-CT, RxNorm)

The pipeline turns raw text not into matched strings, but into a structured representation of linguistic meaning.

1.3 The spaCy Ecosystem for Biomedical Text

spaCy is an industrial-strength NLP library for Python, designed for production use rather than academic experimentation. It delivers fast, accurate linguistic analysis through pre-trained statistical models built on large corpora. For general English text, the en_core_web_sm/md/lg models are the standard starting point.

Medical language presents a domain-specific challenge. General-purpose NLP models are trained predominantly on news articles, books, and web content, corpora that share little with clinical documentation. Medical text is characterized by:

High density of Latin and Greek terminology
Extensive use of abbreviations and acronyms (POD, CABG, LVEF, LIMA)
Domain-specific syntactic patterns
Specialized named entity types (diseases, procedures, anatomical structures, medications)
Negation patterns with clinical significance (“no evidence of”, “without complications”)

This is where scispaCy earns its place. Developed by the Allen Institute for AI, scispaCy provides spaCy-compatible models trained on biomedical scientific literature. The performance gap on clinical text, compared to general-purpose models, is substantial.

2. spaCy Fundamentals for Medical Text

2.1 Installation and Setup

Before proceeding, ensure you have the required libraries installed. spaCy and scispaCy require careful version management:

# Install spaCy
pip install spacy

# Install scispaCy
pip install scispacy

# Install a scispaCy biomedical model
# en_core_sci_md offers a good balance of accuracy and performance
pip install <https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_md-0.5.4.tar.gz>

# For UMLS entity linking (requires additional setup)
pip install scispacy[linker]

# Install spaCy
pip install spacy

# Install scispaCy
pip install scispacy

# Install a scispaCy biomedical model
# en_core_sci_md offers a good balance of accuracy and performance
pip install <https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_md-0.5.4.tar.gz>

# For UMLS entity linking (requires additional setup)
pip install scispacy[linker]

Note on model selection: scispaCy provides several pre-trained models. en_core_sci_sm is the smallest and fastest; en_core_sci_md includes word vectors for better similarity comparisons; en_core_sci_lg offers maximum accuracy. For production medical applications, en_core_sci_md or en_core_sci_lg is recommended. For research requiring UMLS linking specifically, en_core_sci_lg combined with the UmlsEntityLinker is the gold standard.

2.2 The spaCy Document Object Model

When spaCy processes a text string, it returns a Doc object: a rich data structure that exposes all linguistic annotations. Understanding this object model is essential before doing anything serious with the library.

import spacy

# Load the general English model first to understand the basics
# (we will switch to scispaCy models for medical text shortly)
nlp = spacy.load("en_core_web_sm")

sample_text = """
The patient was extubated on POD 1 without complications.
Postoperative atrial fibrillation was noted on POD 3,
successfully cardioverted with amiodarone loading followed by maintenance dose.
"""

doc = nlp(sample_text)

# The Doc object contains Token objects
print(f"Number of tokens:{len(doc)}")

# Each token exposes multiple attributes
for token in doc[:10]:  # First 10 tokens
    print(f"  Token:{token.text!r:20} | POS:{token.pos_:10} | Lemma:{token.lemma_:20} | Stop:{token.is_stop}")

import spacy

# Load the general English model first to understand the basics
# (we will switch to scispaCy models for medical text shortly)
nlp = spacy.load("en_core_web_sm")

sample_text = """
The patient was extubated on POD 1 without complications.
Postoperative atrial fibrillation was noted on POD 3,
successfully cardioverted with amiodarone loading followed by maintenance dose.
"""

doc = nlp(sample_text)

# The Doc object contains Token objects
print(f"Number of tokens:{len(doc)}")

# Each token exposes multiple attributes
for token in doc[:10]:  # First 10 tokens
    print(f"  Token:{token.text!r:20} | POS:{token.pos_:10} | Lemma:{token.lemma_:20} | Stop:{token.is_stop}")

Output:

2.3 Tokenization in Medical Text

Tokenization (splitting text into individual tokens) sounds trivial, but in clinical NLP it is anything but. Take “Amiodarone 200mg”: should this be one token or two? What about “POD 3”, or “LVEF 40%”? Medical abbreviations punctuated with periods (“M.D.”, “i.v.”) add another layer of trouble.

spaCy’s tokenization rules are based on the Penn Treebank standard but can be customized. Let us look at how tokenization handles our discharge summary:

import spacy
from typing import List, Dict

def analyze_tokenization(text: str, nlp) -> List[Dict]:
    """
    Analyze how spaCy tokenizes medical text.

    This function reveals tokenization decisions that are clinically
    significant - for example, whether "100mg" is treated as one token
    or split into "100" and "mg".

    Args:
        text: Clinical text to tokenize
        nlp: Loaded spaCy model

    Returns:
        List of token dictionaries with linguistic attributes
    """
    doc = nlp(text)
    tokens = []

    for token in doc:
        tokens.append({
            'text': token.text,
            'lemma': token.lemma_,
            'pos': token.pos_,          # Coarse-grained POS (NOUN, VERB, etc.)
            'tag': token.tag_,           # Fine-grained POS (NNS, VBD, etc.)
            'dep': token.dep_,           # Syntactic dependency role
            'is_alpha': token.is_alpha,
            'is_digit': token.is_digit,
            'is_stop': token.is_stop,    # Common words with little semantic value
            'is_punct': token.is_punct,
            'shape': token.shape_,       # Character shape: "100mg" → "dddxx"
        })

    return tokens

# Test on the medication section of our discharge summary
medication_text = """
DISCHARGE MEDICATIONS:
1. Aspirin 100mg daily
2. Clopidogrel 75mg daily (for 12 months)
3. Atorvastatin 80mg daily
4. Metoprolol 50mg twice daily
5. Ramipril 5mg daily
6. Amiodarone 200mg daily (for 6 weeks)
"""

# We'll use scispaCy for actual medical processing
nlp_sci = spacy.load("en_core_sci_md")
tokens = analyze_tokenization(medication_text, nlp_sci)

# Print clinically relevant tokens (non-stop, non-punctuation)
print("Clinically Relevant Tokens:")
print(f"{'Text':<20}{'Lemma':<20}{'POS':<10}{'Shape':<15}")
print("-" * 70)
for t in tokens:
    if not t['is_stop'] and not t['is_punct'] and t['text'].strip():
        print(f"{t['text']:<20}{t['lemma']:<20}{t['pos']:<10}{t['shape']:<15}")

import spacy
from typing import List, Dict

def analyze_tokenization(text: str, nlp) -> List[Dict]:
    """
    Analyze how spaCy tokenizes medical text.

    This function reveals tokenization decisions that are clinically
    significant - for example, whether "100mg" is treated as one token
    or split into "100" and "mg".

    Args:
        text: Clinical text to tokenize
        nlp: Loaded spaCy model

    Returns:
        List of token dictionaries with linguistic attributes
    """
    doc = nlp(text)
    tokens = []

    for token in doc:
        tokens.append({
            'text': token.text,
            'lemma': token.lemma_,
            'pos': token.pos_,          # Coarse-grained POS (NOUN, VERB, etc.)
            'tag': token.tag_,           # Fine-grained POS (NNS, VBD, etc.)
            'dep': token.dep_,           # Syntactic dependency role
            'is_alpha': token.is_alpha,
            'is_digit': token.is_digit,
            'is_stop': token.is_stop,    # Common words with little semantic value
            'is_punct': token.is_punct,
            'shape': token.shape_,       # Character shape: "100mg" → "dddxx"
        })

    return tokens

# Test on the medication section of our discharge summary
medication_text = """
DISCHARGE MEDICATIONS:
1. Aspirin 100mg daily
2. Clopidogrel 75mg daily (for 12 months)
3. Atorvastatin 80mg daily
4. Metoprolol 50mg twice daily
5. Ramipril 5mg daily
6. Amiodarone 200mg daily (for 6 weeks)
"""

# We'll use scispaCy for actual medical processing
nlp_sci = spacy.load("en_core_sci_md")
tokens = analyze_tokenization(medication_text, nlp_sci)

# Print clinically relevant tokens (non-stop, non-punctuation)
print("Clinically Relevant Tokens:")
print(f"{'Text':<20}{'Lemma':<20}{'POS':<10}{'Shape':<15}")
print("-" * 70)
for t in tokens:
    if not t['is_stop'] and not t['is_punct'] and t['text'].strip():
        print(f"{t['text']:<20}{t['lemma']:<20}{t['pos']:<10}{t['shape']:<15}")

Clinical Insight on Tokenization: scispaCy handles medical abbreviations more gracefully than general models, although the picture is not uniform. “100mg” may be split or kept together depending on the model and configuration. For downstream extraction tasks, custom tokenization rules for domain-specific patterns are often beneficial:

from spacy.lang.char_classes import ALPHA, ALPHA_LOWER, ALPHA_UPPER
from spacy.lang.en import English

def add_medical_tokenization_rules(nlp):
    """
    Add custom tokenization rules for medical text.

    Medical documents contain patterns that standard tokenizers
    handle poorly: drug-dose combinations, medical acronyms,
    and clinical measurement notations.
    """
    # Prevent splitting on medical dose patterns like "100mg", "5mg"
    # These should remain as single tokens for accurate extraction
    infixes = nlp.Defaults.infixes

    # Add special cases for common medical abbreviations
    special_cases = {
        "POD": [{"ORTH": "POD"}],           # Postoperative Day
        "CPB": [{"ORTH": "CPB"}],           # Cardiopulmonary Bypass
        "CABG": [{"ORTH": "CABG"}],         # Coronary Artery Bypass Grafting
        "LIMA": [{"ORTH": "LIMA"}],         # Left Internal Mammary Artery
        "SVG": [{"ORTH": "SVG"}],           # Saphenous Vein Graft
        "LVEF": [{"ORTH": "LVEF"}],         # Left Ventricular Ejection Fraction
        "LAD": [{"ORTH": "LAD"}],           # Left Anterior Descending
        "RCA": [{"ORTH": "RCA"}],           # Right Coronary Artery
    }

    for text, case in special_cases.items():
        nlp.tokenizer.add_special_case(text, case)

    return nlp

from spacy.lang.char_classes import ALPHA, ALPHA_LOWER, ALPHA_UPPER
from spacy.lang.en import English

def add_medical_tokenization_rules(nlp):
    """
    Add custom tokenization rules for medical text.

    Medical documents contain patterns that standard tokenizers
    handle poorly: drug-dose combinations, medical acronyms,
    and clinical measurement notations.
    """
    # Prevent splitting on medical dose patterns like "100mg", "5mg"
    # These should remain as single tokens for accurate extraction
    infixes = nlp.Defaults.infixes

    # Add special cases for common medical abbreviations
    special_cases = {
        "POD": [{"ORTH": "POD"}],           # Postoperative Day
        "CPB": [{"ORTH": "CPB"}],           # Cardiopulmonary Bypass
        "CABG": [{"ORTH": "CABG"}],         # Coronary Artery Bypass Grafting
        "LIMA": [{"ORTH": "LIMA"}],         # Left Internal Mammary Artery
        "SVG": [{"ORTH": "SVG"}],           # Saphenous Vein Graft
        "LVEF": [{"ORTH": "LVEF"}],         # Left Ventricular Ejection Fraction
        "LAD": [{"ORTH": "LAD"}],           # Left Anterior Descending
        "RCA": [{"ORTH": "RCA"}],           # Right Coronary Artery
    }

    for text, case in special_cases.items():
        nlp.tokenizer.add_special_case(text, case)

    return nlp

2.4 Part-of-Speech Tagging

Part-of-speech (POS) tagging assigns each token its grammatical role. In medical NLP, POS tags are useful for identifying:

Nouns (NOUN, PROPN): Disease names, anatomical structures, drugs
Verbs (VERB): Procedures performed, clinical events (“extubated,” “cardioverted”)
Adjectives (ADJ): Qualitative descriptions (“satisfactory,” “stable,” “reduced”)
Numbers (NUM): Dosages, durations, laboratory values

def extract_clinical_verbs(text: str, nlp) -> List[Dict]:
    """
    Extract action verbs from clinical text.

    In cardiac surgery reports, verbs often encode critical procedural
    and clinical events: "performed," "extubated," "cardioverted,"
    "noted," "removed." This function identifies these for downstream
    event extraction.

    Args:
        text: Clinical narrative text
        nlp: scispaCy model

    Returns:
        List of clinical verbs with their lemmas and surrounding context
    """
    doc = nlp(text)
    clinical_verbs = []

    for token in doc:
        if token.pos_ == "VERB" and not token.is_stop:
            # Get 3-word context window around verb
            start = max(0, token.i - 3)
            end = min(len(doc), token.i + 4)
            context = doc[start:end].text

            clinical_verbs.append({
                'verb': token.text,
                'lemma': token.lemma_,
                'tense': token.tag_,    # VBD = past tense, VBN = past participle
                'context': context,
                'negated': _check_negation(token, doc)  # See negation section
            })

    return clinical_verbs

def _check_negation(token, doc) -> bool:
    """
    Simple negation detection using dependency parsing.

    Checks if a token has a negation modifier in its dependency tree.
    More sophisticated negation handling (like NegEx algorithm) is
    covered in the discussion section.
    """
    for child in token.children:
        if child.dep_ == "neg":
            return True
    return False

# Apply to the postoperative course section
postop_text = """
The patient was extubated on POD 1 without complications.
Chest tubes were removed on POD 2.
Postoperative atrial fibrillation was noted on POD 3,
successfully cardioverted with amiodarone loading.
No wound infections or sternal instability noted.
"""

nlp_sci = spacy.load("en_core_sci_md")
verbs = extract_clinical_verbs(postop_text, nlp_sci)

print("Clinical Verbs Detected:")
print(f"{'Verb':<20}{'Lemma':<20}{'Tense':<8}{'Negated':<10}{'Context'}")
print("-" * 90)
for v in verbs:
    print(f"{v['verb']:<20}{v['lemma']:<20}{v['tense']:<8}{str(v['negated']):<10}{v['context']}")

def extract_clinical_verbs(text: str, nlp) -> List[Dict]:
    """
    Extract action verbs from clinical text.

    In cardiac surgery reports, verbs often encode critical procedural
    and clinical events: "performed," "extubated," "cardioverted,"
    "noted," "removed." This function identifies these for downstream
    event extraction.

    Args:
        text: Clinical narrative text
        nlp: scispaCy model

    Returns:
        List of clinical verbs with their lemmas and surrounding context
    """
    doc = nlp(text)
    clinical_verbs = []

    for token in doc:
        if token.pos_ == "VERB" and not token.is_stop:
            # Get 3-word context window around verb
            start = max(0, token.i - 3)
            end = min(len(doc), token.i + 4)
            context = doc[start:end].text

            clinical_verbs.append({
                'verb': token.text,
                'lemma': token.lemma_,
                'tense': token.tag_,    # VBD = past tense, VBN = past participle
                'context': context,
                'negated': _check_negation(token, doc)  # See negation section
            })

    return clinical_verbs

def _check_negation(token, doc) -> bool:
    """
    Simple negation detection using dependency parsing.

    Checks if a token has a negation modifier in its dependency tree.
    More sophisticated negation handling (like NegEx algorithm) is
    covered in the discussion section.
    """
    for child in token.children:
        if child.dep_ == "neg":
            return True
    return False

# Apply to the postoperative course section
postop_text = """
The patient was extubated on POD 1 without complications.
Chest tubes were removed on POD 2.
Postoperative atrial fibrillation was noted on POD 3,
successfully cardioverted with amiodarone loading.
No wound infections or sternal instability noted.
"""

nlp_sci = spacy.load("en_core_sci_md")
verbs = extract_clinical_verbs(postop_text, nlp_sci)

print("Clinical Verbs Detected:")
print(f"{'Verb':<20}{'Lemma':<20}{'Tense':<8}{'Negated':<10}{'Context'}")
print("-" * 90)
for v in verbs:
    print(f"{v['verb']:<20}{v['lemma']:<20}{v['tense']:<8}{str(v['negated']):<10}{v['context']}")

2.5 Dependency Parsing: Understanding Relationships

Dependency parsing is where linguistic analysis starts to look like genuine clinical intelligence. A dependency parser builds a tree showing how words in a sentence relate to each other grammatically. In medical text, this opens the door to extracting structured relationships:

Subject → Verb → Object: “amiodarone cardioverted atrial fibrillation”
Modifier → Head noun: “postoperative atrial fibrillation” (adjective modifying noun)
Prepositional relationships: “cardioverted with amiodarone” (instrument of action)

def extract_drug_event_relations(text: str, nlp) -> List[Dict]:
    """
    Extract drug-event relationships using dependency parsing.

    Dependency parsing allows us to identify sentences like:
    "successfully cardioverted with amiodarone" →
      drug: amiodarone, action: cardioverted, outcome: successful

    This is a level of extraction impossible with regex alone,
    as it requires understanding sentence structure, not just
    character patterns.

    Args:
        text: Clinical text containing drug and event mentions
        nlp: Loaded spaCy/scispaCy model

    Returns:
        List of extracted drug-event relationships
    """
    doc = nlp(text)
    relations = []

    # Medical procedure verbs relevant to cardiac surgery
    PROCEDURE_VERBS = {
        'cardiovert', 'extubate', 'remove', 'perform',
        'complete', 'administer', 'initiate', 'discontinue'
    }

    # Common cardiac drug keywords
    CARDIAC_DRUGS = {
        'amiodarone', 'aspirin', 'clopidogrel', 'atorvastatin',
        'metoprolol', 'ramipril', 'furosemide', 'metformin'
    }

    for token in doc:
        # Find procedure verbs
        if token.lemma_.lower() in PROCEDURE_VERBS:
            relation = {
                'verb': token.text,
                'verb_lemma': token.lemma_,
                'drug': None,
                'patient_subject': False,
                'outcome_modifier': None,
                'sentence': token.sent.text.strip()
            }

            # Traverse dependency tree for related tokens
            for child in token.children:
                # Check for drug as instrument (prepositional: "with amiodarone")
                if child.dep_ == "prep":
                    for grandchild in child.children:
                        if grandchild.lemma_.lower() in CARDIAC_DRUGS:
                            relation['drug'] = grandchild.text

                # Check for patient as subject ("patient was extubated")
                if child.dep_ in ("nsubj", "nsubjpass"):
                    if child.lemma_.lower() in ("patient",):
                        relation['patient_subject'] = True

                # Check for outcome adverbs ("successfully cardioverted")
                if child.dep_ == "advmod":
                    relation['outcome_modifier'] = child.text

            # Also check the token's own head for drugs
            for sibling in token.head.children:
                if sibling.lemma_.lower() in CARDIAC_DRUGS and sibling != token:
                    if not relation['drug']:
                        relation['drug'] = sibling.text

            if relation['verb']:
                relations.append(relation)

    return relations

# Test on the postoperative section
postop_text = """
The patient was extubated on POD 1 without complications.
Chest tubes were removed on POD 2.
Postoperative atrial fibrillation was noted on POD 3,
successfully cardioverted with amiodarone loading followed by maintenance dose.
"""

relations = extract_drug_event_relations(postop_text, nlp_sci)

print("Drug-Event Relationships Extracted:")
for r in relations:
    print(f"\\n  Verb:{r['verb']}")
    print(f"  Drug:{r['drug'] or 'N/A'}")
    print(f"  Patient subject:{r['patient_subject']}")
    print(f"  Outcome modifier:{r['outcome_modifier'] or 'N/A'}")
    print(f"  Sentence:{r['sentence'][:80]}...")

def extract_drug_event_relations(text: str, nlp) -> List[Dict]:
    """
    Extract drug-event relationships using dependency parsing.

    Dependency parsing allows us to identify sentences like:
    "successfully cardioverted with amiodarone" →
      drug: amiodarone, action: cardioverted, outcome: successful

    This is a level of extraction impossible with regex alone,
    as it requires understanding sentence structure, not just
    character patterns.

    Args:
        text: Clinical text containing drug and event mentions
        nlp: Loaded spaCy/scispaCy model

    Returns:
        List of extracted drug-event relationships
    """
    doc = nlp(text)
    relations = []

    # Medical procedure verbs relevant to cardiac surgery
    PROCEDURE_VERBS = {
        'cardiovert', 'extubate', 'remove', 'perform',
        'complete', 'administer', 'initiate', 'discontinue'
    }

    # Common cardiac drug keywords
    CARDIAC_DRUGS = {
        'amiodarone', 'aspirin', 'clopidogrel', 'atorvastatin',
        'metoprolol', 'ramipril', 'furosemide', 'metformin'
    }

    for token in doc:
        # Find procedure verbs
        if token.lemma_.lower() in PROCEDURE_VERBS:
            relation = {
                'verb': token.text,
                'verb_lemma': token.lemma_,
                'drug': None,
                'patient_subject': False,
                'outcome_modifier': None,
                'sentence': token.sent.text.strip()
            }

            # Traverse dependency tree for related tokens
            for child in token.children:
                # Check for drug as instrument (prepositional: "with amiodarone")
                if child.dep_ == "prep":
                    for grandchild in child.children:
                        if grandchild.lemma_.lower() in CARDIAC_DRUGS:
                            relation['drug'] = grandchild.text

                # Check for patient as subject ("patient was extubated")
                if child.dep_ in ("nsubj", "nsubjpass"):
                    if child.lemma_.lower() in ("patient",):
                        relation['patient_subject'] = True

                # Check for outcome adverbs ("successfully cardioverted")
                if child.dep_ == "advmod":
                    relation['outcome_modifier'] = child.text

            # Also check the token's own head for drugs
            for sibling in token.head.children:
                if sibling.lemma_.lower() in CARDIAC_DRUGS and sibling != token:
                    if not relation['drug']:
                        relation['drug'] = sibling.text

            if relation['verb']:
                relations.append(relation)

    return relations

# Test on the postoperative section
postop_text = """
The patient was extubated on POD 1 without complications.
Chest tubes were removed on POD 2.
Postoperative atrial fibrillation was noted on POD 3,
successfully cardioverted with amiodarone loading followed by maintenance dose.
"""

relations = extract_drug_event_relations(postop_text, nlp_sci)

print("Drug-Event Relationships Extracted:")
for r in relations:
    print(f"\\n  Verb:{r['verb']}")
    print(f"  Drug:{r['drug'] or 'N/A'}")
    print(f"  Patient subject:{r['patient_subject']}")
    print(f"  Outcome modifier:{r['outcome_modifier'] or 'N/A'}")
    print(f"  Sentence:{r['sentence'][:80]}...")

The dependency parser thus enables extraction of structured relationships: not just entity mentions, but how entities interact within clinical events. This capability has direct applications in pharmacovigilance, complication tracking, and automated population of surgical registries.

3. scispaCy: Biomedical NLP

3.1 Why Standard Models Fail for Medical Text

To appreciate scispaCy’s contribution, it helps to compare how a general-purpose spaCy model and a scispaCy model handle the same clinical sentence. Take: “LIMA to LAD anastomosis was completed with satisfactory flow measurements.”

General spaCy (en_core_web_sm):

“LIMA” → labeled as ORG (organization), incorrect
“LAD” → labeled as ORG, incorrect
“anastomosis” → not recognized as a named entity at all

scispaCy (en_core_sci_md):

“LIMA” → recognized with ENTITY label (biomedical entity)
“LAD” → recognized as biomedical entity
“anastomosis” → recognized as biomedical entity

The difference is training data. scispaCy models are trained on the CRAFT corpus (Colorado Richly Annotated Full-Text Corpus) and PubMed abstracts, scientific biomedical text that includes the vocabulary and entity patterns of clinical medicine. The NER annotations in these corpora rely on the BC5CDR schema for chemical and disease entities, plus broader entity annotations from ontologies like UMLS.

3.2 Named Entity Recognition with scispaCy

Named Entity Recognition (NER) is the automatic identification and classification of named entities in text; in our case, medical concepts such as diseases, drugs, procedures, and anatomical structures.

import spacy
from typing import List, Dict, Tuple
import json

# Load scispaCy model
nlp_sci = spacy.load("en_core_sci_md")

# Full discharge summary for NER analysis
DISCHARGE_SUMMARY = """
CARDIAC SURGERY DISCHARGE SUMMARY

Patient: John Smith
MRN: 12345678
DOB: 15/03/1958
Admission Date: 10/01/2024
Discharge Date: 18/01/2024

DIAGNOSIS:
1. Triple vessel coronary artery disease
2. Reduced left ventricular function (LVEF 35%)
3. Hypertension
4. Type 2 Diabetes Mellitus

PROCEDURE PERFORMED:
Coronary Artery Bypass Grafting x3 (CABG)
- LIMA to LAD
- SVG to OM1
- SVG to RCA
Date of surgery: 11/01/2024

OPERATIVE DETAILS:
The procedure was performed via median sternotomy under general anesthesia.
Cardiopulmonary bypass time: 98 minutes
Aortic cross-clamp time: 67 minutes
All anastomoses were completed with satisfactory flow measurements.

POSTOPERATIVE COURSE:
The patient was extubated on POD 1 without complications. Chest tubes were removed on POD 2.
Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone
loading followed by maintenance dose. Patient remained in sinus rhythm thereafter.

No wound infections or sternal instability noted. Echocardiography on POD 5 showed LVEF 40%
with good biventricular function and no pericardial effusion.

DISCHARGE MEDICATIONS:
1. Aspirin 100mg daily
2. Clopidogrel 75mg daily (for 12 months)
3. Atorvastatin 80mg daily
4. Metoprolol 50mg twice daily
5. Ramipril 5mg daily
6. Amiodarone 200mg daily (for 6 weeks)
7. Metformin 1000mg twice daily
8. Furosemide 40mg daily (for 2 weeks)

FOLLOW-UP:
- Outpatient cardiology clinic in 2 weeks
- Cardiac rehabilitation program enrollment
- INR monitoring not required (patient on dual antiplatelet therapy)

The patient was discharged home in stable condition with appropriate wound care instructions.

Dr. Sarah Johnson, MD
Cardiac Surgery Department
"""

def extract_biomedical_entities(text: str, nlp) -> Dict[str, List[str]]:
    """
    Extract all biomedical entities from clinical text using scispaCy NER.

    scispaCy's NER model identifies spans of text corresponding to
    biomedical concepts: diseases, chemicals, genes, cell types,
    species, and more. The entity label set depends on the model used:

    - en_core_sci_*: Uses a single 'ENTITY' label for all biomedical concepts
    - en_ner_bc5cdr_md: Distinguishes DISEASE and CHEMICAL entities
    - en_ner_jnlpba_md: Tags GENE, PROTEIN, RNA, DNA, CELL_TYPE, CELL_LINE

    For general clinical use, en_core_sci_md with UMLS linking provides
    the most complete semantic annotation.

    Args:
        text: Clinical text
        nlp: Loaded scispaCy model

    Returns:
        Dictionary grouping entities by their labels
    """
    doc = nlp(text)

    entities_by_label: Dict[str, List[str]] = {}

    for ent in doc.ents:
        label = ent.label_
        if label not in entities_by_label:
            entities_by_label[label] = []

        # Deduplicate while preserving order
        entity_text = ent.text.strip()
        if entity_text and entity_text not in entities_by_label[label]:
            entities_by_label[label].append(entity_text)

    return entities_by_label

# Extract entities from the discharge summary
entities = extract_biomedical_entities(DISCHARGE_SUMMARY, nlp_sci)

print("Biomedical Entities Extracted by scispaCy:")
print("=" * 60)
for label, ents in sorted(entities.items()):
    print(f"\\n[{label}] ({len(ents)} entities):")
    for entity in ents[:15]:  # Show first 15 per category
        print(f"  -{entity}")

import spacy
from typing import List, Dict, Tuple
import json

# Load scispaCy model
nlp_sci = spacy.load("en_core_sci_md")

# Full discharge summary for NER analysis
DISCHARGE_SUMMARY = """
CARDIAC SURGERY DISCHARGE SUMMARY

Patient: John Smith
MRN: 12345678
DOB: 15/03/1958
Admission Date: 10/01/2024
Discharge Date: 18/01/2024

DIAGNOSIS:
1. Triple vessel coronary artery disease
2. Reduced left ventricular function (LVEF 35%)
3. Hypertension
4. Type 2 Diabetes Mellitus

PROCEDURE PERFORMED:
Coronary Artery Bypass Grafting x3 (CABG)
- LIMA to LAD
- SVG to OM1
- SVG to RCA
Date of surgery: 11/01/2024

OPERATIVE DETAILS:
The procedure was performed via median sternotomy under general anesthesia.
Cardiopulmonary bypass time: 98 minutes
Aortic cross-clamp time: 67 minutes
All anastomoses were completed with satisfactory flow measurements.

POSTOPERATIVE COURSE:
The patient was extubated on POD 1 without complications. Chest tubes were removed on POD 2.
Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone
loading followed by maintenance dose. Patient remained in sinus rhythm thereafter.

No wound infections or sternal instability noted. Echocardiography on POD 5 showed LVEF 40%
with good biventricular function and no pericardial effusion.

DISCHARGE MEDICATIONS:
1. Aspirin 100mg daily
2. Clopidogrel 75mg daily (for 12 months)
3. Atorvastatin 80mg daily
4. Metoprolol 50mg twice daily
5. Ramipril 5mg daily
6. Amiodarone 200mg daily (for 6 weeks)
7. Metformin 1000mg twice daily
8. Furosemide 40mg daily (for 2 weeks)

FOLLOW-UP:
- Outpatient cardiology clinic in 2 weeks
- Cardiac rehabilitation program enrollment
- INR monitoring not required (patient on dual antiplatelet therapy)

The patient was discharged home in stable condition with appropriate wound care instructions.

Dr. Sarah Johnson, MD
Cardiac Surgery Department
"""

def extract_biomedical_entities(text: str, nlp) -> Dict[str, List[str]]:
    """
    Extract all biomedical entities from clinical text using scispaCy NER.

    scispaCy's NER model identifies spans of text corresponding to
    biomedical concepts: diseases, chemicals, genes, cell types,
    species, and more. The entity label set depends on the model used:

    - en_core_sci_*: Uses a single 'ENTITY' label for all biomedical concepts
    - en_ner_bc5cdr_md: Distinguishes DISEASE and CHEMICAL entities
    - en_ner_jnlpba_md: Tags GENE, PROTEIN, RNA, DNA, CELL_TYPE, CELL_LINE

    For general clinical use, en_core_sci_md with UMLS linking provides
    the most complete semantic annotation.

    Args:
        text: Clinical text
        nlp: Loaded scispaCy model

    Returns:
        Dictionary grouping entities by their labels
    """
    doc = nlp(text)

    entities_by_label: Dict[str, List[str]] = {}

    for ent in doc.ents:
        label = ent.label_
        if label not in entities_by_label:
            entities_by_label[label] = []

        # Deduplicate while preserving order
        entity_text = ent.text.strip()
        if entity_text and entity_text not in entities_by_label[label]:
            entities_by_label[label].append(entity_text)

    return entities_by_label

# Extract entities from the discharge summary
entities = extract_biomedical_entities(DISCHARGE_SUMMARY, nlp_sci)

print("Biomedical Entities Extracted by scispaCy:")
print("=" * 60)
for label, ents in sorted(entities.items()):
    print(f"\\n[{label}] ({len(ents)} entities):")
    for entity in ents[:15]:  # Show first 15 per category
        print(f"  -{entity}")

3.3 Specialized NER Models: BC5CDR for Diseases and Chemicals

For clinical NLP requiring a precise distinction between disease entities and chemical/drug entities, the en_ner_bc5cdr_md scispaCy model trained on the BioCreative V CDR corpus is the appropriate choice. The model was trained to distinguish:

DISEASE: Pathological conditions (“atrial fibrillation,” “coronary artery disease,” “hypertension”)
CHEMICAL: Drugs and chemical compounds (“amiodarone,” “aspirin,” “atorvastatin”)

# Note: requires installation of en_ner_bc5cdr_md
# pip install <https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_ner_bc5cdr_md-0.5.4.tar.gz>

def extract_diseases_and_chemicals(text: str) -> Dict[str, List[str]]:
    """
    Use the BC5CDR model to separately identify diseases and drugs.

    The BC5CDR model was trained on 1,500 PubMed articles with
    expert annotations of disease and chemical mentions. It achieves
    F1 scores >85% on the BC5CDR test set, making it suitable for
    clinical applications where chemical-disease distinction matters.

    Clinical application: Automatically extracting the complication
    profile and pharmacological treatment from discharge summaries
    for pharmacovigilance databases.

    Args:
        text: Clinical document text

    Returns:
        Dictionary with 'DISEASE' and 'CHEMICAL' entity lists
    """
    try:
        nlp_cdr = spacy.load("en_ner_bc5cdr_md")
    except OSError:
        print("Model en_ner_bc5cdr_md not installed. Using en_core_sci_md instead.")
        return extract_biomedical_entities(text, spacy.load("en_core_sci_md"))

    doc = nlp_cdr(text)

    result = {"DISEASE": [], "CHEMICAL": []}

    for ent in doc.ents:
        entity_text = ent.text.strip().lower()
        label = ent.label_
        if label in result and entity_text not in result[label]:
            result[label].append(ent.text.strip())

    return result

# Test on our discharge summary
entities_bc5cdr = extract_diseases_and_chemicals(DISCHARGE_SUMMARY)

print("Diseases detected:")
for disease in entities_bc5cdr.get("DISEASE", []):
    print(f"  ✓{disease}")

print("\\nChemicals/Drugs detected:")
for chemical in entities_bc5cdr.get("CHEMICAL", []):
    print(f"  ✓{chemical}")

# Note: requires installation of en_ner_bc5cdr_md
# pip install <https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_ner_bc5cdr_md-0.5.4.tar.gz>

def extract_diseases_and_chemicals(text: str) -> Dict[str, List[str]]:
    """
    Use the BC5CDR model to separately identify diseases and drugs.

    The BC5CDR model was trained on 1,500 PubMed articles with
    expert annotations of disease and chemical mentions. It achieves
    F1 scores >85% on the BC5CDR test set, making it suitable for
    clinical applications where chemical-disease distinction matters.

    Clinical application: Automatically extracting the complication
    profile and pharmacological treatment from discharge summaries
    for pharmacovigilance databases.

    Args:
        text: Clinical document text

    Returns:
        Dictionary with 'DISEASE' and 'CHEMICAL' entity lists
    """
    try:
        nlp_cdr = spacy.load("en_ner_bc5cdr_md")
    except OSError:
        print("Model en_ner_bc5cdr_md not installed. Using en_core_sci_md instead.")
        return extract_biomedical_entities(text, spacy.load("en_core_sci_md"))

    doc = nlp_cdr(text)

    result = {"DISEASE": [], "CHEMICAL": []}

    for ent in doc.ents:
        entity_text = ent.text.strip().lower()
        label = ent.label_
        if label in result and entity_text not in result[label]:
            result[label].append(ent.text.strip())

    return result

# Test on our discharge summary
entities_bc5cdr = extract_diseases_and_chemicals(DISCHARGE_SUMMARY)

print("Diseases detected:")
for disease in entities_bc5cdr.get("DISEASE", []):
    print(f"  ✓{disease}")

print("\\nChemicals/Drugs detected:")
for chemical in entities_bc5cdr.get("CHEMICAL", []):
    print(f"  ✓{chemical}")

3.4 Abbreviation Detection and Expansion

Medical text is dense with abbreviations, a major source of ambiguity for NLP systems. “AF” in a cardiology report almost certainly means atrial fibrillation; in a different context, it might mean acid-fast, amniotic fluid, or audio frequency. scispaCy provides an AbbreviationDetector component that identifies abbreviation-definition pairs within the same document, enabling in-context disambiguation.

import spacy
import scispacy
from scispacy.abbreviation import AbbreviationDetector

def setup_abbreviation_detector(model_name: str = "en_core_sci_md") -> spacy.language.Language:
    """
    Configure scispaCy pipeline with abbreviation detection.

    The AbbreviationDetector implements the algorithm from Schwartz & Hearst (2003),
    which identifies abbreviation-definition pairs based on character matching
    heuristics. When a document defines "CABG" as "Coronary Artery Bypass Grafting",
    the detector enables automatic expansion of all subsequent CABG mentions.

    Returns:
        spaCy language model with abbreviation detection configured
    """
    nlp = spacy.load(model_name)

    # Add abbreviation detector to the pipeline
    nlp.add_pipe("abbreviation_detector")

    return nlp

def extract_abbreviations(text: str, nlp) -> Dict[str, str]:
    """
    Extract and map abbreviations found in clinical text.

    Medical abbreviations are a significant challenge for NLP systems:
    the same abbreviation can have different meanings in different
    specialties or even different sections of the same document.

    This function identifies abbreviation-definition pairs where
    both the abbreviation and its expansion appear in the same document.

    Args:
        text: Clinical document text
        nlp: scispaCy model with AbbreviationDetector

    Returns:
        Dictionary mapping abbreviation → full form
    """
    doc = nlp(text)
    abbreviation_map = {}

    for abbreviation in doc._.abbreviations:
        abbr_text = abbreviation.text
        # The long form is the identified full expansion
        long_form = abbreviation._.long_form.text if abbreviation._.long_form else "Unknown"
        abbreviation_map[abbr_text] = long_form

    return abbreviation_map

# Note: Our discharge summary uses abbreviations but may not always define them inline
# Let's create a version with explicit definitions to demonstrate the capability
annotated_summary = """
The patient underwent Coronary Artery Bypass Grafting (CABG) using Left Internal
Mammary Artery (LIMA) to Left Anterior Descending (LAD) and Saphenous Vein Graft (SVG)
to Obtuse Marginal (OM1) and Right Coronary Artery (RCA).
Cardiopulmonary Bypass (CPB) time was 98 minutes.
Postoperative atrial fibrillation (AF) was noted and treated with amiodarone.
Left Ventricular Ejection Fraction (LVEF) improved from 35% to 40%.
"""

nlp_with_abbrev = setup_abbreviation_detector()
abbreviations = extract_abbreviations(annotated_summary, nlp_with_abbrev)

print("Abbreviations detected and expanded:")
for abbr, expansion in sorted(abbreviations.items()):
    print(f"{abbr} →{expansion}")

import spacy
import scispacy
from scispacy.abbreviation import AbbreviationDetector

def setup_abbreviation_detector(model_name: str = "en_core_sci_md") -> spacy.language.Language:
    """
    Configure scispaCy pipeline with abbreviation detection.

    The AbbreviationDetector implements the algorithm from Schwartz & Hearst (2003),
    which identifies abbreviation-definition pairs based on character matching
    heuristics. When a document defines "CABG" as "Coronary Artery Bypass Grafting",
    the detector enables automatic expansion of all subsequent CABG mentions.

    Returns:
        spaCy language model with abbreviation detection configured
    """
    nlp = spacy.load(model_name)

    # Add abbreviation detector to the pipeline
    nlp.add_pipe("abbreviation_detector")

    return nlp

def extract_abbreviations(text: str, nlp) -> Dict[str, str]:
    """
    Extract and map abbreviations found in clinical text.

    Medical abbreviations are a significant challenge for NLP systems:
    the same abbreviation can have different meanings in different
    specialties or even different sections of the same document.

    This function identifies abbreviation-definition pairs where
    both the abbreviation and its expansion appear in the same document.

    Args:
        text: Clinical document text
        nlp: scispaCy model with AbbreviationDetector

    Returns:
        Dictionary mapping abbreviation → full form
    """
    doc = nlp(text)
    abbreviation_map = {}

    for abbreviation in doc._.abbreviations:
        abbr_text = abbreviation.text
        # The long form is the identified full expansion
        long_form = abbreviation._.long_form.text if abbreviation._.long_form else "Unknown"
        abbreviation_map[abbr_text] = long_form

    return abbreviation_map

# Note: Our discharge summary uses abbreviations but may not always define them inline
# Let's create a version with explicit definitions to demonstrate the capability
annotated_summary = """
The patient underwent Coronary Artery Bypass Grafting (CABG) using Left Internal
Mammary Artery (LIMA) to Left Anterior Descending (LAD) and Saphenous Vein Graft (SVG)
to Obtuse Marginal (OM1) and Right Coronary Artery (RCA).
Cardiopulmonary Bypass (CPB) time was 98 minutes.
Postoperative atrial fibrillation (AF) was noted and treated with amiodarone.
Left Ventricular Ejection Fraction (LVEF) improved from 35% to 40%.
"""

nlp_with_abbrev = setup_abbreviation_detector()
abbreviations = extract_abbreviations(annotated_summary, nlp_with_abbrev)

print("Abbreviations detected and expanded:")
for abbr, expansion in sorted(abbreviations.items()):
    print(f"{abbr} →{expansion}")

Expected output:

Abbreviations detected and expanded:
  AF → atrial fibrillation
  CABG → Coronary Artery Bypass Grafting
  CPB → Cardiopulmonary Bypass
  LAD → Left Anterior Descending
  LIMA → Left Internal Mammary Artery
  LVEF → Left Ventricular Ejection Fraction
  OM1 → Obtuse Marginal
  RCA → Right Coronary Artery
  SVG → Saphenous Vein Graft

A caveat from clinical practice: the Schwartz & Hearst algorithm only catches abbreviations that are explicitly defined inside the same document. Most operative reports we read in our daily work assume the reader already knows what CABG, LIMA or LVEF mean and never spell them out. For real deployment, this detector almost always needs to be paired with a curated cardiac-surgery glossary as a fallback.

4. UMLS Entity Linking: Connecting to Medical Ontologies

4.1 The Importance of Standardized Medical Vocabulary

Named entity recognition tells us that a text span refers to a medical concept; UMLS linking tells us which standardized concept it refers to. The distinction matters for interoperability.

Consider: “coronary artery disease,” “CAD,” “ischemic heart disease,” and “arteriosclerotic heart disease” are all strings pointing to the same underlying condition. In the Unified Medical Language System (UMLS), this concept has a single Concept Unique Identifier: C0010068. By linking extracted entities to UMLS CUIs we get:

Semantic normalization: All equivalent surface forms map to the same concept
Cross-database interoperability: UMLS bridges SNOMED-CT, ICD-10, MeSH, RxNorm, and thousands of other vocabularies
Knowledge graph integration: CUIs can be used to query external medical knowledge bases
Research reproducibility: Studies using CUI-based cohort definitions are unambiguous

The UMLS Metathesaurus is maintained by the U.S. National Library of Medicine and covers over 3.5 million concepts across more than 200 biomedical vocabularies. Key semantic type hierarchies relevant to cardiac surgery include:

T047, Disease or Syndrome (coronary artery disease, atrial fibrillation)
T121, Pharmacologic Substance (amiodarone, aspirin)
T061, Therapeutic or Preventive Procedure (CABG, cardioversion)
T023, Body Part, Organ, or Organ Component (left ventricle, coronary artery)

4.2 Implementing UMLS Entity Linking with scispaCy

import spacy
import scispacy
from scispacy.linking import EntityLinker

def setup_umls_pipeline(model_name: str = "en_core_sci_md") -> spacy.language.Language:
    """
    Configure scispaCy pipeline with UMLS entity linking.

    The EntityLinker component compares extracted entity spans against
    a local copy of the UMLS knowledge base using approximate string
    matching and concept embeddings. It returns ranked candidate CUIs
    with confidence scores.

    IMPORTANT: The UMLS linker requires ~2GB RAM for the knowledge base
    and has a one-time download of ~1.5GB. Performance is substantially
    better with en_core_sci_lg than with the smaller models.

    Configuration parameters:
    - resolve_abbreviations: Use AbbreviationDetector before linking
    - linker_name: "umls" uses full UMLS; "mesh" uses MeSH only (smaller)
    - threshold: Minimum similarity score (0.7-0.85 recommended for clinical use)
    - max_entities_per_mention: Candidates per entity span

    Returns:
        Configured spaCy pipeline with UMLS linking
    """
    nlp = spacy.load(model_name)

    # Add abbreviation detection (must come before linker)
    nlp.add_pipe("abbreviation_detector")

    # Add UMLS entity linker
    nlp.add_pipe(
        "scispacy_linker",
        config={
            "resolve_abbreviations": True,       # Expand abbreviations before linking
            "linker_name": "umls",               # Full UMLS knowledge base
            "threshold": 0.80,                   # Minimum similarity threshold
            "max_entities_per_mention": 3,       # Top-k candidates
        }
    )

    return nlp

def extract_umls_entities(text: str, nlp) -> List[Dict]:
    """
    Extract entities with full UMLS annotations.

    For each recognized biomedical entity, this function retrieves:
    - The entity text as it appears in the document
    - The UMLS Concept Unique Identifier (CUI)
    - The canonical (preferred) name for the concept
    - The UMLS semantic type (T047 = Disease, T121 = Drug, etc.)
    - The similarity score (confidence of the linking)
    - The definition from UMLS (if available)

    Args:
        text: Clinical document text
        nlp: Pipeline with UMLS linker configured

    Returns:
        List of entities with complete UMLS annotations
    """
    doc = nlp(text)

    # Access the linker component for knowledge base queries
    linker = nlp.get_pipe("scispacy_linker")

    annotated_entities = []

    for ent in doc.ents:
        entity_info = {
            'text': ent.text,
            'start_char': ent.start_char,
            'end_char': ent.end_char,
            'label': ent.label_,
            'umls_candidates': []
        }

        # Get UMLS linking results
        for umls_entity in ent._.kb_ents:
            cui = umls_entity[0]         # UMLS Concept Unique Identifier
            score = umls_entity[1]       # Similarity score (0-1)

            # Query the knowledge base for this CUI
            if cui in linker.kb.cui_to_entity:
                kb_entity = linker.kb.cui_to_entity[cui]
                candidate = {
                    'cui': cui,
                    'canonical_name': kb_entity.canonical_name,
                    'aliases': list(kb_entity.aliases[:5]),        # First 5 aliases
                    'types': list(kb_entity.types),                # Semantic type codes
                    'definition': kb_entity.definition or "No definition available",
                    'similarity_score': round(score, 4)
                }
                entity_info['umls_candidates'].append(candidate)

        if entity_info['umls_candidates']:
            annotated_entities.append(entity_info)

    return annotated_entities

def format_umls_report(entities: List[Dict]) -> str:
    """
    Format UMLS entity linking results as a readable clinical report.

    Args:
        entities: Output from extract_umls_entities()

    Returns:
        Formatted string report
    """
    lines = ["UMLS Entity Linking Report", "=" * 60]

    # Semantic type descriptions for readability
    SEMANTIC_TYPES = {
        "T047": "Disease or Syndrome",
        "T121": "Pharmacologic Substance",
        "T061": "Therapeutic/Preventive Procedure",
        "T023": "Body Part, Organ, Component",
        "T116": "Amino Acid/Peptide/Protein",
        "T048": "Mental/Behavioral Dysfunction",
        "T033": "Finding",
        "T184": "Sign or Symptom",
        "T060": "Diagnostic Procedure",
    }

    for entity in entities:
        lines.append(f"\\nEntity: '{entity['text']}'")
        lines.append(f"  Character span: [{entity['start_char']}:{entity['end_char']}]")

        if entity['umls_candidates']:
            best = entity['umls_candidates'][0]  # Top candidate
            lines.append(f"  Best UMLS match:")
            lines.append(f"    CUI:{best['cui']}")
            lines.append(f"    Canonical name:{best['canonical_name']}")
            lines.append(f"    Similarity:{best['similarity_score']:.2%}")

            # Describe semantic types
            type_descriptions = [
                SEMANTIC_TYPES.get(t, t) for t in best['types']
            ]
            lines.append(f"    Semantic types:{', '.join(type_descriptions)}")

            if best['aliases']:
                lines.append(f"    Known aliases:{', '.join(best['aliases'][:3])}")

    return "\\n".join(lines)

# Example usage
# Note: requires ~2GB RAM for UMLS knowledge base
# nlp_umls = setup_umls_pipeline()
# entities = extract_umls_entities(DISCHARGE_SUMMARY, nlp_umls)
# print(format_umls_report(entities))

import spacy
import scispacy
from scispacy.linking import EntityLinker

def setup_umls_pipeline(model_name: str = "en_core_sci_md") -> spacy.language.Language:
    """
    Configure scispaCy pipeline with UMLS entity linking.

    The EntityLinker component compares extracted entity spans against
    a local copy of the UMLS knowledge base using approximate string
    matching and concept embeddings. It returns ranked candidate CUIs
    with confidence scores.

    IMPORTANT: The UMLS linker requires ~2GB RAM for the knowledge base
    and has a one-time download of ~1.5GB. Performance is substantially
    better with en_core_sci_lg than with the smaller models.

    Configuration parameters:
    - resolve_abbreviations: Use AbbreviationDetector before linking
    - linker_name: "umls" uses full UMLS; "mesh" uses MeSH only (smaller)
    - threshold: Minimum similarity score (0.7-0.85 recommended for clinical use)
    - max_entities_per_mention: Candidates per entity span

    Returns:
        Configured spaCy pipeline with UMLS linking
    """
    nlp = spacy.load(model_name)

    # Add abbreviation detection (must come before linker)
    nlp.add_pipe("abbreviation_detector")

    # Add UMLS entity linker
    nlp.add_pipe(
        "scispacy_linker",
        config={
            "resolve_abbreviations": True,       # Expand abbreviations before linking
            "linker_name": "umls",               # Full UMLS knowledge base
            "threshold": 0.80,                   # Minimum similarity threshold
            "max_entities_per_mention": 3,       # Top-k candidates
        }
    )

    return nlp

def extract_umls_entities(text: str, nlp) -> List[Dict]:
    """
    Extract entities with full UMLS annotations.

    For each recognized biomedical entity, this function retrieves:
    - The entity text as it appears in the document
    - The UMLS Concept Unique Identifier (CUI)
    - The canonical (preferred) name for the concept
    - The UMLS semantic type (T047 = Disease, T121 = Drug, etc.)
    - The similarity score (confidence of the linking)
    - The definition from UMLS (if available)

    Args:
        text: Clinical document text
        nlp: Pipeline with UMLS linker configured

    Returns:
        List of entities with complete UMLS annotations
    """
    doc = nlp(text)

    # Access the linker component for knowledge base queries
    linker = nlp.get_pipe("scispacy_linker")

    annotated_entities = []

    for ent in doc.ents:
        entity_info = {
            'text': ent.text,
            'start_char': ent.start_char,
            'end_char': ent.end_char,
            'label': ent.label_,
            'umls_candidates': []
        }

        # Get UMLS linking results
        for umls_entity in ent._.kb_ents:
            cui = umls_entity[0]         # UMLS Concept Unique Identifier
            score = umls_entity[1]       # Similarity score (0-1)

            # Query the knowledge base for this CUI
            if cui in linker.kb.cui_to_entity:
                kb_entity = linker.kb.cui_to_entity[cui]
                candidate = {
                    'cui': cui,
                    'canonical_name': kb_entity.canonical_name,
                    'aliases': list(kb_entity.aliases[:5]),        # First 5 aliases
                    'types': list(kb_entity.types),                # Semantic type codes
                    'definition': kb_entity.definition or "No definition available",
                    'similarity_score': round(score, 4)
                }
                entity_info['umls_candidates'].append(candidate)

        if entity_info['umls_candidates']:
            annotated_entities.append(entity_info)

    return annotated_entities

def format_umls_report(entities: List[Dict]) -> str:
    """
    Format UMLS entity linking results as a readable clinical report.

    Args:
        entities: Output from extract_umls_entities()

    Returns:
        Formatted string report
    """
    lines = ["UMLS Entity Linking Report", "=" * 60]

    # Semantic type descriptions for readability
    SEMANTIC_TYPES = {
        "T047": "Disease or Syndrome",
        "T121": "Pharmacologic Substance",
        "T061": "Therapeutic/Preventive Procedure",
        "T023": "Body Part, Organ, Component",
        "T116": "Amino Acid/Peptide/Protein",
        "T048": "Mental/Behavioral Dysfunction",
        "T033": "Finding",
        "T184": "Sign or Symptom",
        "T060": "Diagnostic Procedure",
    }

    for entity in entities:
        lines.append(f"\\nEntity: '{entity['text']}'")
        lines.append(f"  Character span: [{entity['start_char']}:{entity['end_char']}]")

        if entity['umls_candidates']:
            best = entity['umls_candidates'][0]  # Top candidate
            lines.append(f"  Best UMLS match:")
            lines.append(f"    CUI:{best['cui']}")
            lines.append(f"    Canonical name:{best['canonical_name']}")
            lines.append(f"    Similarity:{best['similarity_score']:.2%}")

            # Describe semantic types
            type_descriptions = [
                SEMANTIC_TYPES.get(t, t) for t in best['types']
            ]
            lines.append(f"    Semantic types:{', '.join(type_descriptions)}")

            if best['aliases']:
                lines.append(f"    Known aliases:{', '.join(best['aliases'][:3])}")

    return "\\n".join(lines)

# Example usage
# Note: requires ~2GB RAM for UMLS knowledge base
# nlp_umls = setup_umls_pipeline()
# entities = extract_umls_entities(DISCHARGE_SUMMARY, nlp_umls)
# print(format_umls_report(entities))

Expected output for key entities (illustrative, actual CUIs from UMLS 2024AB):

UMLS Entity Linking Report

Entity: ‘coronary artery disease’
Character span: [150:172]
Best UMLS match:
CUI: C0010068
Canonical name: Coronary Arteriosclerosis
Similarity: 96.80%
Semantic types: Disease or Syndrome
Known aliases: CAD, Ischemic Heart Disease, Coronary Artery Disease

Entity: ‘atrial fibrillation’
Character span: [478:496]
Best UMLS match:
CUI: C0004238
Canonical name: Atrial Fibrillation
Similarity: 99.10%
Semantic types: Disease or Syndrome, Finding
Known aliases: AF, A-fib, Auricular Fibrillation

Entity: ‘amiodarone’
Character span: [521:531]
Best UMLS match:
CUI: C0002598
Canonical name: Amiodarone
Similarity: 99.90%
Semantic types: Pharmacologic Substance
Known aliases: Cordarone, Nexterone, Pacerone

Entity: ‘median sternotomy’
Best UMLS match:
CUI: C0185792
Canonical name: Median Sternotomy
Similarity: 98.50%
Semantic types: Therapeutic/Preventive Procedure

5. Temporal Information Extraction

5.1 The Clinical Importance of Temporal Relations

In cardiac surgery, the timing of events carries weight comparable to the events themselves. “Atrial fibrillation on POD 3” and “atrial fibrillation on POD 14” may imply different etiologies, different risks, and different management approaches. A complete NLP pipeline therefore needs to capture not only what happened, but when.

spaCy’s dependency parser is well suited to extracting temporal relationships:

def extract_temporal_clinical_events(text: str, nlp) -> List[Dict]:
    """
    Extract clinical events with their temporal anchors.

    Uses dependency parsing to identify relationships between
    clinical events and their associated postoperative day (POD),
    time expressions, or date references.

    Temporal patterns in cardiac surgery discharge summaries:
    - Explicit POD references: "extubated on POD 1"
    - Implicit timing: "chest tubes removed subsequently"
    - Duration: "amiodarone for 6 weeks"
    - Relative timing: "48 hours after surgery"

    Args:
        text: Clinical narrative text
        nlp: scispaCy model

    Returns:
        List of events with temporal information
    """
    doc = nlp(text)

    # Pattern for "POD N", postoperative day
    pod_pattern = re.compile(r'POD\\s*(\\d+)', re.IGNORECASE)

    events_with_timing = []

    for sent in doc.sents:
        sent_text = sent.text

        # Find POD references in this sentence
        pod_matches = pod_pattern.findall(sent_text)
        pod_days = [int(d) for d in pod_matches]

        # Find clinical events (non-stop verbs and nouns)
        clinical_tokens = []
        for token in sent:
            if (token.pos_ in ("VERB", "NOUN", "PROPN") and
                not token.is_stop and
                len(token.text) > 2):
                clinical_tokens.append(token.lemma_)

        if pod_days and clinical_tokens:
            events_with_timing.append({
                'sentence': sent_text.strip(),
                'pod_days': pod_days,
                'earliest_pod': min(pod_days),
                'clinical_tokens': clinical_tokens[:8],  # Top 8 content words
            })

    # Sort by earliest POD
    events_with_timing.sort(key=lambda x: x['earliest_pod'])

    return events_with_timing

# Build clinical timeline from the postoperative course
postop_text = """
The patient was extubated on POD 1 without complications. Chest tubes were removed on POD 2.
Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone
loading followed by maintenance dose. Patient remained in sinus rhythm thereafter.
Echocardiography on POD 5 showed LVEF 40% with good biventricular function.
"""

timeline = extract_temporal_clinical_events(postop_text, nlp_sci)

print("Clinical Timeline Reconstruction:")
print("=" * 60)
for event in timeline:
    print(f"\\n  POD{event['earliest_pod']}:")
    print(f"{event['sentence']}")

def extract_temporal_clinical_events(text: str, nlp) -> List[Dict]:
    """
    Extract clinical events with their temporal anchors.

    Uses dependency parsing to identify relationships between
    clinical events and their associated postoperative day (POD),
    time expressions, or date references.

    Temporal patterns in cardiac surgery discharge summaries:
    - Explicit POD references: "extubated on POD 1"
    - Implicit timing: "chest tubes removed subsequently"
    - Duration: "amiodarone for 6 weeks"
    - Relative timing: "48 hours after surgery"

    Args:
        text: Clinical narrative text
        nlp: scispaCy model

    Returns:
        List of events with temporal information
    """
    doc = nlp(text)

    # Pattern for "POD N", postoperative day
    pod_pattern = re.compile(r'POD\\s*(\\d+)', re.IGNORECASE)

    events_with_timing = []

    for sent in doc.sents:
        sent_text = sent.text

        # Find POD references in this sentence
        pod_matches = pod_pattern.findall(sent_text)
        pod_days = [int(d) for d in pod_matches]

        # Find clinical events (non-stop verbs and nouns)
        clinical_tokens = []
        for token in sent:
            if (token.pos_ in ("VERB", "NOUN", "PROPN") and
                not token.is_stop and
                len(token.text) > 2):
                clinical_tokens.append(token.lemma_)

        if pod_days and clinical_tokens:
            events_with_timing.append({
                'sentence': sent_text.strip(),
                'pod_days': pod_days,
                'earliest_pod': min(pod_days),
                'clinical_tokens': clinical_tokens[:8],  # Top 8 content words
            })

    # Sort by earliest POD
    events_with_timing.sort(key=lambda x: x['earliest_pod'])

    return events_with_timing

# Build clinical timeline from the postoperative course
postop_text = """
The patient was extubated on POD 1 without complications. Chest tubes were removed on POD 2.
Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone
loading followed by maintenance dose. Patient remained in sinus rhythm thereafter.
Echocardiography on POD 5 showed LVEF 40% with good biventricular function.
"""

timeline = extract_temporal_clinical_events(postop_text, nlp_sci)

print("Clinical Timeline Reconstruction:")
print("=" * 60)
for event in timeline:
    print(f"\\n  POD{event['earliest_pod']}:")
    print(f"{event['sentence']}")

Output:

Clinical Timeline Reconstruction:
============================================================

  POD 1:
  The patient was extubated on POD 1 without complications.

  POD 2:
  Chest tubes were removed on POD 2.

  POD 3:
  Postoperative atrial fibrillation was noted on POD 3, successfully cardioverted with amiodarone loading.

  POD 5:
  Echocardiography on POD 5 showed LVEF 40% with good biventricular function.

This kind of structured temporal reconstruction has direct applications in automated quality metrics reporting, particularly for surgical safety indicators like time-to-extubation and time-to-complication onset.

6. Comparing spaCy/scispaCy with Regex: A Decision Framework

6.1 Quantitative Performance Considerations

Neither regex nor linguistic NLP is universally better. The optimal choice depends on the specific extraction task, the consistency of the source documents, and the computational resources you actually have at your disposal. The table below summarizes the practical considerations:

Criterion	Regex	spaCy/scispaCy
Setup complexity	Minimal	Moderate (model installation)
Processing speed	Very fast (~1ms/doc)	Moderate (100ms–5s/doc)
Memory requirements	Negligible	500MB–2GB
Accuracy for structured fields	High (when patterns are known)	Moderate
Accuracy for free-text narrative	Low	High
Negation handling	Manual, brittle	Built-in via dependency parsing
Semantic normalization	None	UMLS linking
Maintenance burden	High (pattern updates)	Low (model updates)
Explainability	Complete	Partial (neural models)
Regulatory compliance	Straightforward	Requires validation

6.2 When to Use Regex

Regex is still the preferred approach for:

Highly structured fields with predictable formats: dates, MRN numbers, medication dosages in formatted lists
Real-time applications where processing latency is critical
Resource-constrained environments (edge computing, embedded systems)
Deterministic requirements where output must be fully explainable and reproducible
Simple extraction rules with few variations (e.g., “extract all numbers followed by ‘mg’”)

6.3 When to Use spaCy/scispaCy

Linguistic NLP earns its place when you need:

Free-text narratives with high syntactic variability (postoperative course sections)
Entity recognition where surface forms vary (“AF,” “atrial fib,” “atrial fibrillation”)
Relationship extraction between medical concepts (drug → indication, procedure → complication)
Negation detection at scale (well above what rule-based approaches can deliver)
Interoperability requirements where UMLS/SNOMED-CT standardization is needed
Research applications requiring semantic search or concept-based cohort selection

6.4 The Hybrid Pipeline (Preview)

In practice, the most robust systems combine both. A hybrid pipeline for cardiac surgery discharge summaries might follow this logic:

Regex layer: Extract structured fields (dates, numeric values, section headers) with high precision
scispaCy NER layer: Identify medical entities in free-text sections
UMLS linking layer: Normalize entities to standard vocabularies
Validation layer: Cross-validate regex and NLP results; flag discrepancies

The architecture is explored in depth in Article 3, where we layer in machine learning classifiers for complication severity scoring and LLM-based reasoning for the harder inference tasks.

7. Conclusion: From Syntax to Semantics

This article has shown how spaCy and scispaCy push medical text analysis past character-level pattern matching, into something closer to genuine linguistic intelligence. The key advances over regex-based extraction are:

Semantic awareness: scispaCy NER recognizes medical concepts regardless of their surface form, handling the terminological variability inherent to clinical language. “Triple vessel coronary artery disease,” “3-vessel CAD,” and “severe multivessel coronary disease” all refer to the same clinical condition.

Relational extraction: Dependency parsing exposes the grammatical relationships between tokens, enabling extraction of drug-event associations, procedural outcomes, and temporal sequences that regex simply cannot reach.

Ontological grounding: UMLS entity linking standardizes extracted concepts to universal identifiers, supporting interoperability across electronic health record systems, clinical registries, and research databases. Once a CUI is assigned to a concept, that annotation is unambiguous across institutions and across time.

Negation sensitivity: Medical NLP that cannot distinguish affirmed from negated findings is, plainly, clinically dangerous. The dependency-based negation detection illustrated here, while not exhaustive, handles the majority of common patterns in discharge documentation.

Limitations of the Linguistic Approach

scispaCy, biomedical training notwithstanding, is not without limitations that need to be acknowledged before any clinical use:

Out-of-vocabulary terms: Novel drug names, rare procedures, or institution-specific abbreviations may not be recognized
Domain shift: Models trained on PubMed abstracts may underperform on clinical notes, which have different linguistic characteristics
Negation complexity: Advanced negation patterns (“unlikely to represent,” “cannot exclude”) require specialized components (MedSpaCy, NegEx) beyond standard dependency parsing
Numerical relationship extraction: With “LVEF 35%”, scispaCy recognizes LVEF as an entity but does not inherently link the numerical value; that gap is best filled by regex or a post-processing layer
Validation requirements: Clinical deployment of NLP systems requires rigorous validation on institution-specific documents, with performance metrics (precision, recall, F1) computed on expert-annotated gold standards

8. Looking Ahead: Machine Learning and LLMs

The third and final article in this series will tackle what neither regex nor linguistic NLP can do: learn from data, handle unrestricted text variation, and perform complex clinical inference.

Machine learning classifiers trained on annotated discharge summaries can pick up complication risk from subtle linguistic patterns. Large Language Models (LLMs), the Claude API among them, can extract arbitrarily complex information through natural language querying, compare findings against clinical guidelines, and generate structured summaries from narrative text. The article will also address the non-negotiable topics: GDPR/HIPAA compliance, de-identification pipelines, and deployment considerations for production medical AI systems.

Preview: scispaCy might recognize “atrial fibrillation” as a disease entity. A properly prompted LLM, by contrast, can determine whether the documented management (amiodarone loading, cardioversion) was consistent with current ACC/AHA guidelines, a level of clinical reasoning that pattern-based NLP systems cannot reach.

References

Neumann M, et al. “ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing.” Proceedings of the 18th BioNLP Workshop and Shared Task, 2019:319–327.
Honnibal M, Montani I. “spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.” Unpublished, 2017.
Bodenreider O. “The Unified Medical Language System (UMLS): integrating biomedical terminology.” Nucleic Acids Research 2004;32(suppl_1):D267–D270.
Schwartz AS, Hearst MA. “A simple algorithm for identifying abbreviation definitions in biomedical text.” Proceedings of the Pacific Symposium on Biocomputing, 2003:451–462.
Li J, et al. “BioCreative V CDR task corpus: a resource for chemical disease relation extraction.” Database 2016:baw068.
Lample G, et al. “Neural Architectures for Named Entity Recognition.” Proceedings of NAACL-HLT 2016, 2016:260–270.
Chapman WW, et al. “A simple algorithm for identifying negated findings and diseases in discharge summaries.” Journal of Biomedical Informatics 2001;34(5):301–310. [NegEx algorithm]
Soldaini L, Goharian N. “QuickUMLS: a fast, unsupervised approach for medical concept extraction.” MedIR Workshop, SIGIR, 2016.
Peng Y, et al. “Transfer Learning in Biomedical Natural Language Processing.” Proceedings of the 18th BioNLP Workshop, 2019.
Savova GK, et al. “Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications.” JAMIA 2010;17(5):507–513.
Uzuner O, et al. “2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text.” JAMIA 2011;18(5):552–556.
Stenetorp P, et al. “BRAT: a Web-based Tool for NLP-Assisted Text Annotation.” Proceedings of EACL 2012, 2012:102–107.