Computer Vision That Actually Ships: Notes from the Deployment Side

April 2018. The FDA clears IDx-DR — an AI that reads retinal images for diabetic retinopathy and spits out a screening decision without a clinician in the loop. That's the first autonomous AI diagnostic cleared in any field of medicine. The pivotal trial behind it: 900 patients across 10 primary care sites. I keep coming back to that number because it's the kind of evidence that doesn't show up on a leaderboard.

I've spent enough time around CV projects to know the pattern. Team builds something clever. Paper gets written. Benchmark number goes up. Then someone asks: where does this run? On what hardware? Who's accountable when it's wrong? And the room gets quiet.

AlexNet in 2012 is the usual origin story, and it's true that it nearly halved object recognition error, which kicked off the deep learning era. But AlexNet didn't matter because it won ILSVRC. It mattered because people could suddenly build things that worked.

Where it actually counts

Medical imaging is the clearest case. The JAMA 2016 study trained a CNN on 128,175 retinal images and hit an AUC of 0.991 on EyePACS-1 and 0.990 on Messidor-2. Those aren't parlor tricks — that's a system that can flag disease in a rural clinic where the nearest ophthalmologist is two hours away.

U-Net is another one I keep recommending. It was built for biomedical segmentation, won the ISBI cell tracking challenge in 2015, and segments a 512x512 image in under a second on a GPU. Small model, narrow problem, real utility. That combination is underrated.

Autonomous driving is where the deployment pressure gets brutal. nuScenes was the first dataset with a full sensor suite — 6 cameras, 5 radars, 1 lidar, 360-degree coverage, 1000 scenes at 20 seconds each, with 7x the annotations and 100x the images of KITTI. That richness exists for a reason: a car has to handle the weird stuff. A pedestrian in a Halloween costume at dusk. A mattress on the freeway. No curated benchmark prepares you for that.

But wait — benchmarks aren't useless

I'll defend ImageNet and COCO here. ImageNet's 14 million images across 21,841 synsets fueled the whole deep learning boom, and ILSVRC pulled in 50+ institutions a year at its peak. COCO's 2.5 million labeled instances across 328,000 images pushed detection and segmentation forward. ResNet won ILSVRC 2015 at 3.57% error and became the backbone for wins in detection, localization, and segmentation. None of that happens without benchmarks.

The problem isn't benchmarks. It's treating them as the destination. Once a benchmark saturates, the marginal gain from another 0.3% is worth almost nothing in production.

What I actually watch now is efficiency. EfficientDet-D7 gets 55.1 AP on COCO with 77M parameters and 410B FLOPs — 4x to 9x smaller and 13x to 42x fewer FLOPs than earlier detectors. YOLOv4 does 43.5% AP at around 65 FPS on a Tesla V100. Those numbers matter because they determine whether your model runs on a $200 edge device or needs a $10,000 cloud GPU. That's the difference between a product and a demo.

What I'd actually do

If you're starting a CV project, here's the shape of it:

Notice none of those rows say "use the latest transformer." ViT and Swin are genuinely impressive — Swin hits 87.3 top-1 on ImageNet-1K — but they often need massive pre-training data and compute. For a lot of applications, a tuned CNN or something like MobileNetV2 is the right answer.

Rough sequence I'd follow:

• Name the operational metric first. False negatives in screening. End-to-end latency in driving. Not accuracy in the abstract.
• Pick a dataset that looks like your deployment. Cityscapes for urban driving. Kinetics for action recognition. If your real data doesn't look like the training data, you're already in trouble.
• Start with something proven. YOLO for speed. Faster R-CNN for accuracy. U-Net for segmentation.
• Spend the time on data quality and augmentation. U-Net leans on strong augmentation and can train from surprisingly few images.

One thing nobody tells you: before you fine-tune a giant model, check whether a smaller one fits your latency budget. EfficientNet-B7 gets 84.3% top-1 on ImageNet while being 8.4x smaller and 6.1x faster than the previous best ConvNet. I've watched that gap decide whether a feature ships on-device or gets stuck in the cloud forever.

A caveat worth saying out loud: a model that's 2% worse on a benchmark but 10x faster and runs offline is usually the better product. The 2% rarely shows up in user-visible outcomes. The latency always does.

So don't pick an architecture because it's winning a leaderboard. Pick the simplest thing that solves the actual problem, then ship it, measure it, and iterate. That's the whole game.

Sources

• IDx-DR pivotal trial - https://pubmed.ncbi.nlm.nih.gov/31304320/
• JAMA 2016 deep learning diabetic retinopathy study - https://pubmed.ncbi.nlm.nih.gov/27898976/
• U-Net paper - https://arxiv.org/abs/1505.04597
• nuScenes dataset paper - https://arxiv.org/abs/1903.11027
• EfficientDet paper - https://arxiv.org/abs/1911.09070
• YOLOv4 paper - https://arxiv.org/abs/2004.10934