Note accuracy is the foundational metric for AI scribe evaluation and the one with the most direct patient safety implications. It measures the degree to which an AI-generated clinical note faithfully represents the actual clinical encounter — specifically whether it introduces content that was not present in the visit (hallucination) or omits content that was (omission). Both failure types carry distinct risks: hallucinated medications or dosages create prescribing hazards; omitted findings may cause documentation gaps that affect downstream care decisions or malpractice exposure.

Measurement requires a ground truth corpus of annotated visit pairs: a human-transcribed or EHR-verified record of what actually occurred, matched against the AI-generated note. Annotation categories typically include medication mentions (name, dose, frequency, route), diagnosis statements, exam findings, lab values, and plan elements. Each discrepancy is classified as a hallucination, omission, or paraphrase-equivalent. The resulting metric is usually expressed as an error rate per note segment (e.g., medication hallucination rate per 100 note-medications) rather than a single aggregate score, since different error types carry different clinical weights.

Published evidence on this metric is sparse and methodologically inconsistent. Most AI scribe vendors report internal accuracy figures without releasing annotation schemas or inter-rater reliability statistics, making cross-vendor comparison impossible. A small number of academic studies have examined LLM hallucination in clinical note generation — notably work by Umapathi et al. (2023) on medical hallucinations in GPT-4 and subsequent evaluations using the MedHallu framework — but these use synthetic or de-identified corpora that may not reflect real encounter complexity. To our knowledge, no published study has evaluated hallucination rate across multiple frontier models on matched, specialty-stratified real-world clinical encounters using a standardized annotation schema. This study aims to address that gap.

The AI scribe pipeline begins before the LLM. For most deployed systems, a spoken clinical encounter is first transcribed by an automatic speech recognition (ASR) model, and only then processed by the language model into a structured note. Errors at the transcription stage propagate forward: a misheard drug name or an incorrect dosage in the transcript will be reproduced — sometimes confidently — in the generated note. Standard Word Error Rate (WER) is an insufficient metric in this context because it weights all words equally. Transcribing "the patient reports fatigue" as "the patient reports failure" is a WER of one substitution — but transcribing "carvedilol 25 mg twice daily" as "carvedilol 25 mg daily" is also one substitution, with a fundamentally different consequence.

Medical WER extends the standard metric by applying domain-specific weights to a curated vocabulary of high-stakes terms: drug names, dosage units, medical procedures, anatomical landmarks, and numerical values associated with clinical thresholds. Some implementations further distinguish between low-stakes paraphrasing errors and semantically consequential substitutions. Character Error Rate (CER) is additionally useful for evaluating transcription of drug names where single-character differences produce entirely different medications (e.g., hydroxyzine vs. hydralazine).

The clinical AI literature has generated a growing body of ASR evaluation work, particularly since the proliferation of ambient documentation systems. Lybarger et al. evaluated ASR performance specifically on clinical notes across specialties; Miner et al. (2020) characterized WER on medical dictation across multiple commercial ASR platforms. However, most published evaluations were conducted before the current generation of large ASR models (Whisper, Azure Cognitive Services, Google Medical Speech) and may not reflect current system performance. Additionally, few studies stratify by speaker accent, which is clinically relevant for international healthcare settings.

End-to-end latency measures the elapsed time from the end of a clinical encounter recording to the moment a complete, reviewable note is available to the provider. In deployed ambulatory settings, this typically must fall under 60–90 seconds to avoid disrupting clinical workflow — a provider seeing 20–25 patients in a day cannot absorb multi-minute delays per encounter without schedule impact. Latency is therefore a hard feasibility constraint as much as a performance dimension.

The metric is best reported at p50 (median) and p95 (95th percentile). Median latency reflects typical performance; the p95 value captures the tail behavior that determines whether the system is reliably usable — a scribe that averages 30 seconds but occasionally takes 4 minutes creates unpredictable provider experience. Latency decomposes into ASR processing time, LLM inference time (including time-to-first-token and token generation rate), and any orchestration overhead. For streaming-capable models, time-to-first-token may matter as much as total generation time, since providers can begin reviewing partial notes before generation completes.

Published latency benchmarks specific to clinical AI are rare; most vendor disclosures omit p95 statistics and report under controlled rather than production-load conditions. General LLM inference benchmarks (e.g., Artificial Analysis) provide useful comparative data on token throughput and TTFT across model providers, but these measure raw generation speed rather than the end-to-end clinical pipeline. Our evaluation protocol measures latency from recording upload to first rendered note character, under standardized network conditions, across visit lengths of 10, 20, and 45 minutes.

Clinical AI scribes are not used with a fixed prompt. Providers establish note preferences — preferred section order, narrative versus bullet format for HPI, how to handle medication reconciliation, whether to include verbatim patient quotes — that should persist across encounters. Instruction following rate measures the proportion of provider-specified formatting and content preferences that are honored in generated notes, evaluated across repeated generations with the same instruction set. It is a measure of reliability rather than capability: a model that can produce a psychiatry MSE correctly when explicitly instructed, but drops it 30% of the time on subsequent regenerations, is not clinically usable for that workflow.

Measurement requires a defined preference set — a structured list of provider-specified instructions across formatting, content inclusion, and structural conventions — evaluated against generated notes by a combination of rule-based checkers (for deterministic preferences like section presence) and human annotators (for stylistic preferences like narrative tone). General-purpose instruction following benchmarks such as IFEval (Zhou et al., 2023) exist, but they test simple, verifiable constraints in short-context settings that do not reflect the complexity of multi-preference clinical note generation. No published benchmark evaluates instruction following specifically in the context of AI scribe customization.

Most clinical AI evaluations are conducted on short, controlled clinical vignettes that occupy a fraction of a model's nominal context window. Real clinical encounters are substantially longer: a complex new patient visit in pediatric endocrinology may involve 40–50 minutes of recorded conversation generating 8,000–14,000 tokens of transcribed text before the LLM processes it. The "lost in the middle" phenomenon — in which LLM attention degrades for content appearing in the middle of long contexts relative to content near the beginning or end — has been documented across multiple model families (Liu et al., 2023). Its specific impact on clinical note generation has not been systematically characterized.

This metric is evaluated by comparing note accuracy (using the hallucination/omission framework from §4.1) stratified by visit length: short (<15 min), medium (15–30 min), and long (>30 min). Accuracy degradation is expressed as the relative change in error rate from the short to the long visit cohort. A clinically meaningful degradation threshold has not been formally established; we propose that a >20% relative increase in hallucination rate from short to long visits constitutes a clinically significant finding warranting disclosure in deployment documentation.

Clinical documentation conventions differ substantially across specialties, and an aggregate accuracy score obscures performance variation that matters enormously in deployment decisions. A psychiatric progress note requires a structured mental status exam (orientation, affect, thought process, cognition, insight), suicide risk assessment language anchored to specific frameworks (Columbia Suicide Severity Rating Scale), and safety planning documentation — none of which appear in a primary care SOAP note. A pediatric endocrinology visit requires growth parameters (height, weight, BMI percentile, pubertal staging), insulin regimen documentation with basal-bolus ratios, and CGM metrics. An OB/GYN encounter centers on gestational age, fundal height, fetal heart tones, and trimester-specific risk counseling documentation.

General-purpose LLMs trained primarily on broad medical corpora may produce structurally plausible notes for primary care that nonetheless fail the specialty-specific documentation requirements that determine reimbursement, regulatory compliance, and continuity of care. Specialty-stratified evaluation requires separate annotation schemas and ground truth corpora for each specialty of interest. Our initial evaluation targets three specialties available through our clinical data partnership: pediatric endocrinology (n ≈ 150 encounters), psychiatry (n ≈ 120 encounters), and primary care (n ≈ 200 encounters). Each specialty has a distinct rubric developed in consultation with board-certified clinicians in that field.

Independent of content accuracy, clinical notes must conform to structural conventions that serve documentation, reimbursement, and legal functions. The SOAP format (Subjective, Objective, Assessment, Plan) is the standard framework in ambulatory care, but its implementation varies by specialty, institution, and payer requirements. Structure fidelity evaluates whether generated notes contain required sections, whether content appears in the correct section (a physical exam finding placed in the Subjective section is a structural error regardless of accuracy), and whether section-specific conventions are honored (e.g., Assessment sections should contain diagnostic statements with supporting rationale, not isolated lab values).

This metric is partially automatable: section presence and order can be verified programmatically, while content placement and section-internal conventions require human review or a secondary LLM judge. A related but distinct quality dimension is note length calibration — notes that are excessively verbose reduce provider efficiency and may bury clinically important information; notes that are too brief may fail E/M coding requirements that determine reimbursement level. Published work by Gao et al. (2023) on automated SOAP note evaluation provides a relevant methodological foundation, though it does not include the multi-model comparison and specialty stratification proposed here.