Biomedical computer vision looks straightforward on a slide: feed a medical image into a convolutional neural network, get back a diagnosis, a segmentation mask, or a flagged region. In practice, the pipelines that survive contact with a hospital are shaped less by model architecture than by the constraints around it — image acquisition variance, regulatory artefacts, PACS and EHR integration, and the operational discipline of running a model that radiologists and surgeons depend on. This piece walks through what those constraints actually look like, and where the engineering work concentrates. The framing we use across our engagements: a biomedical CV system is not a model. It is an inference path with traceable inputs, version-locked weights, monitored outputs, and a clinical workflow it does not disrupt. Everything else — the architecture, the loss function, the augmentation strategy — is downstream of that. What Does a Production Biomedical Computer Vision Pipeline Actually Contain? Strip away the marketing language and a clinical-grade CV pipeline has six layers worth naming explicitly. Layer What it does Where it usually breaks Acquisition normalisation Harmonises MRI, CT, ultrasound, or pathology slides across scanner vendors, field strengths, and protocols Scanner-specific intensity distributions cause silent accuracy drops Preprocessing Resampling, bias-field correction, intensity normalisation, registration Inconsistent voxel spacing across sites breaks segmentation networks Inference Classification, segmentation (U-Net family), detection (RetinaNet, nnU-Net, transformer-based variants) Out-of-distribution scans degrade silently rather than failing loudly Post-processing Connected-component analysis, lesion measurement, uncertainty estimation Naive thresholding produces unstable measurements run-to-run Integration DICOM I/O, PACS routing, EHR write-back, HL7/FHIR messaging Custom integrations consume more engineering time than the model itself Monitoring Drift detection, calibration tracking, exception sampling for clinician review Drift surfaces months after deployment, after referral patterns have already shifted Each of these is its own engineering surface. We see practitioners underestimate the acquisition-normalisation layer in particular: a model trained on Siemens 3T MRI data behaves differently on GE 1.5T data not because the network is fragile, but because the input distribution genuinely shifts. This is an observed pattern across our medical-imaging engagements, not a benchmarked rate — but it shows up reliably enough that we now treat scanner-vendor coverage as a planning variable, not a footnote. Segmentation, Detection, Classification: Why The Mix Matters The three CV task classes used in biomedical applications carry different validation profiles, and conflating them is one of the more common mistakes in early-stage programmes. Classification answers a yes/no or multi-class question on a whole image — does this chest X-ray show pneumonia, does this dermatology image show suspected melanoma. It is the easiest to validate against ground truth because labels are discrete. It is also the easiest to over-trust, because a single number hides where the model is looking. Detection locates objects within an image — a nodule on a CT scan, a polyp on a colonoscopy frame, a microcalcification on a mammogram. Detection is where computer-aided detection (CADe) systems live. Validation requires bounding-box overlap metrics (mAP at IoU thresholds), and the failure mode that matters clinically is missed detections at low-prevalence operating points. Segmentation produces pixel-level masks — tumour volume, organ boundaries, vessel trees. This is the substrate for treatment planning (radiation dose calculation, surgical path planning) and for longitudinal disease tracking. U-Net and its descendants remain the workhorse architecture; nnU-Net’s contribution was less a new architecture than a disciplined recipe for hyperparameter and preprocessing selection across medical-imaging tasks. Production systems almost always combine these. A radiology assist might use detection to flag candidate regions, classification to grade malignancy probability, and segmentation to measure lesion size for longitudinal comparison. The integration of the three is where most of the engineering complexity actually lives. How Does Medical Computer Vision Handle Generalisability and Drift? Generalisability is the single most-discussed and most-misunderstood property of clinical CV systems. A model that achieves AUC 0.95 on its training site’s held-out test set is not, on that basis, a model that will achieve AUC 0.95 at a different hospital. The distribution shift can come from scanner hardware, reconstruction kernels, patient demographics, protocol differences, or all of these at once. The practical response splits into three pieces: Multi-site training data, not just multi-site test data. Models trained on data from a single institution generalise poorly, regardless of training-set size. The diversity of acquisition conditions matters more than raw image count past a certain threshold. Domain adaptation at inference time, when retraining is not feasible. Histogram matching, style transfer, and test-time normalisation can recover meaningful accuracy on out-of-distribution scans without touching model weights — which matters because under FDA SaMD rules, touching weights is a regulatory event. Post-market drift monitoring that compares the distribution of inference-time inputs (and outputs) to the validation cohort. Drift in input statistics often precedes accuracy degradation; catching it early lets the team trigger a planned retraining and re-validation cycle rather than discovering the problem through clinician escalation. The lock-and-key constraint deserves explicit attention: under SaMD rules, a cleared model is approved as a specific weight configuration. Drift mitigation cannot quietly update weights in production. This is one of the structural differences between medical-device CV and consumer-grade CV — and it shapes pipeline design from the first sprint, not the regulatory submission. Integration With PACS, EHR, And Clinical Workflow The model is the small part. Integration is the larger part. A CV system that cannot read DICOM cleanly, route results back to PACS, write structured findings into the EHR, and surface flags inside the clinician’s existing reading workflow is a research artefact, not a clinical tool. Common patterns we see in production: DICOM routing as the contract surface. The CV pipeline subscribes to DICOM SCP endpoints, receives studies as they arrive, and returns secondary capture series or structured reports (DICOM SR) tagged with the originating study. This keeps the model out of the radiologist’s main workflow until a flag is raised. HL7 / FHIR messaging for EHR write-back. Quantitative findings (lesion volumes, longitudinal change measurements) land as discrete observations the EHR can query, not as PDF reports the EHR cannot parse. Worklist prioritisation rather than autonomous triage. Early-generation systems tried to triage cases away from clinicians. Current-generation systems reorder the reading worklist so suspected-positive cases surface first, with the clinician still reading every study. This is both clinically safer and a softer regulatory posture. These patterns are not exotic, but they are where most of the production engineering time actually goes. The CV model is often two months of the project; the integration scaffolding is six. Surgical Precision And Real-Time Inference Real-time CV in surgery — instrument tracking, tissue segmentation during minimally invasive procedures, augmented-reality overlays for surgical guidance — sits at the intersection of low-latency inference and clinical-grade accuracy. The engineering constraint is sharper than diagnostic imaging: latencies above roughly 100 ms break the surgeon’s perceptual loop, and dropped frames are not acceptable. Production stacks for this typically rely on TensorRT or ONNX Runtime for optimised inference, CUDA-native preprocessing pipelines, and careful management of GPU memory across the imaging-acquisition and inference stages. The model architecture choice is often constrained more by latency budget than by accuracy ceiling — a slightly less accurate detector that runs at 60 fps will outperform a more accurate one that runs at 12 fps, because the surgical workflow cannot wait for the second one. Pre-operative planning uses the same model families differently. Segmentation of organ structures from CT or MRI produces 3D models surgeons can rehearse against, and the constraint shifts from latency to anatomical fidelity. The same underlying CV stack supports both modes — what changes is the validation evidence and the deployment target. What Are The Operational Constraints Specific To Biomedical CV? Three operational realities distinguish biomedical CV from general computer vision work: Data privacy and residency. Medical images are protected health information. Training data pipelines need de-identification (often beyond DICOM header stripping — pixel-level burned-in PHI is common), and inference systems need to operate within the data-residency boundaries of the institution. This rules out many off-the-shelf cloud inference patterns and pushes deployments toward on-premise or hospital-VPC architectures. Annotation cost and reliability. Pixel-level annotations from board-certified radiologists are slow and expensive. Inter-rater variability is real — two radiologists annotating the same lesion will often disagree on boundaries by clinically meaningful margins. Production systems plan annotation budgets and consensus protocols from day one, not when training accuracy plateaus. Audit-grade traceability. Every inference in a cleared device must be traceable to the model version that produced it, the input image hash, and the post-processing parameters applied. This is not a logging detail; it is a regulatory requirement that shapes how the pipeline records and stores intermediate artefacts. These constraints are why biomedical CV programmes that design for FDA validation evidence from day one — rather than retrofitting compliance at submission time — tend to reach cleared-device status materially faster than programmes that optimise for accuracy first. The Engineering Thread That Connects These Biomedical computer vision is not a single technology stack. It is a family of pipelines that share a common discipline: input normalisation, version-locked inference, integration with existing clinical infrastructure, and post-deployment monitoring. The model is one component among several. The programmes that succeed treat it that way from the first design review. For the regulatory-pathway view of the same problem — how these engineering choices translate into FDA SaMD evidence and cleared-device economics — see AI-Enabled Medical Devices: The Computer Vision Layer Behind FDA-Cleared Tools. For the deep-learning-specific architecture walkthrough, see Deep Learning in Medical Computer Vision: How It Works. For broader programme context across our engagements, our Computer Vision R&D practice covers the operational thread end to end. FAQ How many AI-enabled medical devices has the FDA cleared, and which CV patterns recur across them? The FDA maintains a public list of AI/ML-enabled medical devices that runs into the high hundreds and continues to grow. The recurring CV patterns across cleared devices cluster around radiology assist (detection and triage on CT, MRI, mammography, chest X-ray), ophthalmology screening (diabetic retinopathy grading from fundus images), dermatology triage (lesion classification), and pathology (whole-slide image analysis). Most cleared devices target a specific anatomy and a specific clinical question — generalist medical CV models remain rare in cleared-device status. What are the production patterns behind FDA-cleared CV diagnostics (CADe, CADx, radiomics)? CADe (computer-aided detection) systems flag candidate regions for clinician review. CADx (computer-aided diagnosis) systems add a probability or category to flagged findings. Radiomics pipelines extract quantitative features — texture, shape, intensity distributions — from segmented regions for downstream prognostic modelling. In production, these are usually layered: a detection model surfaces candidates, a classification head grades them, and a radiomics layer produces measurements for longitudinal tracking. How does deep learning in medical CV (classification, segmentation, detection) translate into regulatory artefacts? Each task class maps to different validation evidence. Classification needs reader studies comparing model output against clinician consensus, usually with ROC/AUC analysis at clinically relevant operating points. Detection requires sensitivity and specificity reporting against verified ground truth, often with free-response ROC (FROC) curves. Segmentation needs Dice coefficient and Hausdorff distance against expert annotations, with inter-rater variability characterised. All three need pre-specified statistical analysis plans submitted before the validation study runs. Where do AI medical-device pipelines need to handle generalisability, drift, and population shift? Generalisability is handled at training time through multi-site, multi-scanner data and at inference time through domain-adaptation techniques. Drift is handled through post-market surveillance that monitors input distributions and output rates against the validation cohort. Population shift — when the patient mix at a deployment site differs from the training population — is the hardest of the three to detect and the most likely to surface as a slow accuracy decline rather than a sharp failure. What integration patterns connect CV inference to PACS, EHR, and clinical workflow? DICOM SCP/SCU endpoints for image ingest and result routing, DICOM Structured Reports or secondary capture series for findings, and HL7 or FHIR messages for EHR write-back of quantitative observations. Worklist prioritisation — reordering the reading queue rather than triaging cases away — is the dominant deployment posture for current-generation systems. Which AI-enabled medical-device companies and products define the current state of practice in 2026? The cleared-device landscape spans large imaging vendors (GE, Siemens, Philips) with embedded AI features, dedicated AI-imaging companies (Aidoc, Annalise.ai, Viz.ai, Arterys, Paige.AI in pathology), and ophthalmology specialists (IDx-DR was the first autonomous AI diagnostic cleared). The state of practice is defined less by individual products than by the regulatory frameworks they operate under — FDA SaMD in the US, MDR / IVDR in Europe — and the validation-evidence patterns those frameworks now expect.
