How to Choose a Computer Vision Model for Your Application

Your model has 87.8% top-1 accuracy on ImageNet, but your application runs on a $99 phone. That's the gap I see constantly. Computer vision isn't about squeezing out the last percentage point on a leaderboard—it's about matching an architecture to the constraints of a real deployment. I'm writing this for engineers and technical founders who need to pick a model, train it, and ship it without burning six months. My blunt recommendation: start with a pre-trained model, fine-tune on your data, and only then consider architectural surgery. Here's how to do it in five steps.

1. Define the task before you name the model

Classification gives you a label. Detection gives you a label and a box. Segmentation gives you a label and a pixel mask. If you confuse these, you'll waste weeks. For example, if you're building a system to flag diabetic retinopathy from retinal scans, you only need classification—a single label per image. The IDx-DR system became the first FDA-authorized autonomous AI diagnostic in any field of medicine in April 2018, based on a pivotal trial of 900 patients at 10 primary care sites (IDx-DR pivotal trial). That's classification. But if you're building an autonomous vehicle that must avoid pedestrians, you need detection and tracking. The KITTI benchmark, captured around Karlsruhe, includes up to 15 cars and 30 pedestrians per image (KITTI Vision Benchmark Suite). You can't solve that with a classifier.

2. Match the architecture to your latency budget

I've seen teams pick a heavy two-stage detector for a real-time robot and then act surprised when it crawls. Don't be that team. Two-stage detectors like Faster R-CNN with VGG-16 run at 5 fps on a GPU using only 300 proposals per image (Faster R-CNN paper). That's fine for offline analysis. But if you need 30 fps on an embedded device, you need a one-stage detector. YOLO processes images at 45 frames per second, and its smaller variant, Fast YOLO, hits 155 fps (YOLO paper). The trade-off? YOLO makes more localization errors but far fewer false detections. So if you're building a security camera that alerts on intruders, false alarms are expensive—YOLO's bias might actually help. But if you're measuring a part's position to within a millimeter, you need the two-stage precision.

3. Pick a backbone that fits your data and compute

Backbones are the feature extractors. ResNet introduced residual learning to train networks up to 152 layers—8x deeper than VGG but with lower complexity (Deep Residual Learning paper). That depth gave a 28% relative improvement on COCO object detection. If you have limited data, don't start from scratch. Use a pre-trained model. For mobile, MobileNetV2 uses inverted residual blocks to keep inference efficient. For segmentation, U-Net can be trained from very few images and segments a 512x512 image in under a second on a recent GPU (U-Net paper). And if you have a massive dataset and want the best accuracy, Swin Transformer reaches 87.3 top-1 accuracy on ImageNet-1K, 58.7 box AP on COCO, and 53.5 mIoU on ADE20K (Swin Transformer paper). But that comes at a compute cost you may not have.

4. Validate with the right metric—not just accuracy

Accuracy alone will lie to you when classes are imbalanced. Use precision, recall, and Intersection over Union (IoU). For detection, IoU tells you how well your predicted box overlaps the ground truth. For medical screening, you need high sensitivity. In the JAMA 2016 diabetic retinopathy study, at a high-sensitivity operating point the algorithm achieved 97.5% sensitivity and 93.4% specificity on EyePACS-1, and 96.1% sensitivity and 93.9% specificity on Messidor-2. That means it missed fewer cases but raised more false alarms. Your choice depends on the cost of a miss versus a false alarm. I recommend you plot the precision-recall curve and pick the operating point that matches your clinical or business need—not the default threshold.

5. What can go wrong: adversarial examples and distribution shift

Your model is not a magic oracle. Small, intentionally worst-case perturbations—adversarial examples—can cause neural networks to misclassify with high confidence (Explaining and Harnessing Adversarial Examples paper). If you deploy a face recognition system, for instance, an attacker could wear a printed pattern to fool it. And distribution shift will kill you: a model trained on clean ImageNet images may fail on blurry, low-light photos from a cheap camera. Always test on data from your actual deployment environment. I've seen a model with 99% validation accuracy drop to 60% in the field because the lighting changed.

Comparison: Which model family for which application?

That table is a starting point, not gospel. EfficientDet-D7, for example, reaches 55.1 AP on COCO with 77M parameters—4x to 9x smaller and using 13x to 42x fewer FLOPs than previous detectors (EfficientDet paper). But if you're on a drone with a 10-watt power budget, even that might be too much. You'll need to quantize or prune.

The single most important thing to remember: your model's accuracy on a benchmark is meaningless if it doesn't meet your latency, memory, and cost constraints in the real world. Start with a pre-trained model, fine-tune on your data, measure on your target hardware, and only then optimize. Ship something that works, not something that wins a leaderboard.

Sources

• YOLO paper - https://arxiv.org/abs/1506.02640
• Faster R-CNN paper - https://arxiv.org/abs/1506.01497
• U-Net paper - https://arxiv.org/abs/1505.04597
• JAMA 2016 deep learning diabetic retinopathy study - https://pubmed.ncbi.nlm.nih.gov/27898976/
• Explaining and Harnessing Adversarial Examples paper - https://arxiv.org/abs/1412.6572