Explainability in computer vision is usually sold as an ethics control. In practice, it is mostly a debugging tool — and the teams that get the most out of it treat it that way first, and as a fairness instrument second. A saliency map that tells you why a tumour classifier fired on the corner watermark instead of the lesion is the same map that tells you why the model is biased: it is looking at the wrong pixels. That reframe matters because the dominant narrative around XAI conflates two different jobs: explaining a single prediction and auditing a model’s behaviour across a population. The methods that do one well are not the methods that do the other well, and confusing them is how teams end up shipping models that pass an ethics review and then fail in deployment. What XAI in computer vision is actually doing The standard toolkit divides into a few families, and each answers a different question: Method What it answers Where it earns its keep Grad-CAM / Grad-CAM++ “Which spatial regions drove this CNN’s class score?” Fast sanity check during training; catches shortcut learning SHAP (DeepSHAP, GradientSHAP) “What is each input feature’s marginal contribution?” Tabular metadata fused with image features; medical imaging audits LIME “Which superpixels, if removed, change the prediction?” Per-instance review with non-ML stakeholders Attention rollout / attention maps “Which patches did the ViT attend to?” Transformer-based detectors and segmentation models Concept-based (TCAV, ACE) “Does the model rely on a human-named concept?” Regulated domains where features must be nameable None of these methods give you ground truth. They give you a story consistent with the model’s gradients or perturbation responses. The story is useful, but it is a story. Why this matters for production In our experience across vision engagements, the first hour with Grad-CAM on a freshly trained classifier almost always reveals something embarrassing: the model is keying on a hospital’s scanner watermark, a timestamp burned into the corner of a CCTV frame, or the consistent background colour of a particular product photo studio. This is the observed pattern that justifies XAI as a standard part of the training loop — not a compliance afterthought. The two jobs XAI does, and why teams confuse them Job 1 — Per-prediction explanation A radiologist looking at a single chest X-ray needs to know why the model called it pneumonia on this patient. Grad-CAM or LIME on that one image, overlaid on the original, gives them a region. They can agree, disagree, or escalate. The unit of analysis is one prediction. Job 2 — Population-level audit A regulator asking “does this facial recognition system misidentify darker-skinned women at a higher rate?” needs aggregated statistics across thousands of predictions, broken down by demographic slice. Per-image saliency maps are nearly useless here. What you actually want is disaggregated error analysis — precision, recall, and false-positive rates per slice — supplemented by concept-level checks (TCAV-style) to see whether the model is using legitimate features. The methods used for Job 1 are not the right instruments for Job 2. A model can produce perfectly reasonable-looking Grad-CAMs on every individual prediction and still be systematically biased at the population level. Teams that only run per-instance XAI and skip the slice-based statistical audit are doing fairness theatre. Where XAI actively misleads A few failure modes show up often enough to call out: Saliency confirmation bias. A pretty heatmap over the “right” region of an image feels like proof the model is reasoning correctly. It is not. The model may have learned a spurious correlation that happens to live in the same spatial area as the legitimate signal. Grad-CAM cannot distinguish “looking at the lesion because it’s a lesion” from “looking at the lesion because lesions co-occur with a particular acquisition artefact in that quadrant”. LIME instability. LIME’s superpixel perturbations are stochastic. Run it twice on the same image and you can get visibly different explanations. For per-instance review this is tolerable; for audit evidence it is not. Attention is not explanation. This is a well-known caveat in the transformer literature, and it applies to vision transformers too. Attention weights tell you what the model attended to, not what it used. Two attention maps can produce the same prediction; the same attention map can produce different predictions under small perturbations. Edge deployment loses the explanation budget. A model running on a Jetson or a phone NPU rarely has the compute headroom to also run SHAP at inference time. Teams either log enough state to reconstruct explanations offline, or they accept that production explanations will be coarser than training-time ones. A practical XAI workflow for a CV team The workflow we recommend on most engagements: During training: Grad-CAM (or its variants) on a sample of validation predictions, every epoch, as a sanity dashboard. The goal is to catch shortcut learning early, before a model with the right loss curve and the wrong reasoning ships. Before release: A structured slice analysis — per-class, per-demographic-slice (where applicable), per-acquisition-condition. This is the population audit. Pair it with concept-based checks if the domain has nameable concepts (skin tone, lighting condition, pose). In production: Log enough input metadata and intermediate activations to reconstruct explanations offline when a prediction is contested. Real-time XAI at the edge is usually not worth the latency tax. For stakeholder review: LIME or SHAP overlays on a curated sample, with the caveat that these are “consistent stories”, not proofs. The connecting tissue is annotation discipline. None of these methods produce useful signal if the underlying labels are noisy or inconsistent — a model trained on sloppy bounding boxes will produce sloppy saliency, and you cannot debug your way out of that with XAI. How does XAI interact with regulation? The EU AI Act and similar frameworks ask for meaningful information about the logic involved, not a specific technique. That phrasing is deliberate. A regulator is unlikely to be satisfied with a folder of Grad-CAM PNGs; they want documentation of the model’s intended use, training data composition, evaluation slices, and known failure modes. XAI tooling supports that documentation — it does not substitute for it. This is the practitioner reframe: XAI is one input into a transparency story, alongside model cards, datasheets, slice-based evaluation, and human review processes. Teams that try to make XAI carry the entire compliance load end up with thin documentation and brittle defences. Where this sits relative to the broader CV stack Explainability is a cross-cutting concern, not a stage. It touches data labelling, training, evaluation, deployment, and post-deployment monitoring. For a team mapping out a production CV system from first principles, it sits alongside the other quality concerns covered in our practitioner-tuned beginner’s guide to computer vision fundamentals — not as a separate phase, but as a discipline applied at every phase. The honest summary: XAI in computer vision is genuinely useful, mostly as a debugging and slice-audit tool, and it does not deliver the things its strongest advocates claim. Treat it as one instrument in a panel, calibrate to the question you are actually asking, and document what it cannot tell you. FAQ What are the five stages of computer vision from acquisition to inference, and where does engineering effort concentrate? Acquisition, preprocessing, feature extraction, model inference, and post-processing/integration. Engineering effort concentrates disproportionately on the first and last stages — acquisition conditions and integration with downstream systems — even though tutorials spend most of their pages on the middle three. XAI fits into evaluation, which sits across all five. How does computer vision work end-to-end in a 2026 production stack? A typical stack ingests frames from cameras or stored media, runs preprocessing (normalisation, resize, sometimes calibration), passes tensors through a CNN or transformer-based model — often via TensorRT, ONNX Runtime, or a vendor SDK — and emits structured outputs that feed business logic. XAI hooks attach at the model stage for debugging and at the evaluation stage for audits. Which language (Python vs C++) fits which CV workload, and why is that no longer a religious debate? Python dominates training, experimentation, and most XAI tooling. C++ dominates latency-critical inference, embedded deployment, and integration into existing native applications. The debate softened because PyTorch and ONNX Runtime expose comparable ergonomics on both sides, and bindings are stable enough that the choice is now driven by deployment target, not preference. What separates a CV practitioner from a CV researcher in deliverables and tooling? Researchers ship papers, benchmarks, and reference implementations. Practitioners ship systems that hold up under acquisition drift, edge-device constraints, and stakeholder review. The tooling diverges accordingly: practitioners spend more time on data pipelines, monitoring, and XAI-as-debugging than on novel architectures. Where do the canonical CV textbooks (Szeliski, Nixon, Forsyth) still hold up, and where do they need refresh? The classical geometry, image formation, and feature-engineering chapters remain solid foundations. The deep-learning chapters age fastest — transformer-based vision, self-supervised pretraining, and modern XAI methods are either underweighted or absent. Pair the textbooks with current survey papers. What is the minimal foundation needed to ship a production CV system in a real engineering team? A clear problem framing (classification vs detection vs segmentation vs tracking), a labelled dataset that reflects deployment conditions, a baseline model, a slice-based evaluation protocol, an XAI-supported debugging loop, and a deployment target with known latency and memory budgets. Most failures we see come from skipping the slice-based evaluation, not from picking the wrong model.
