Skip the Model Zoo: Pick One Detector and Ship It

Stop shopping for models. The biggest killer of computer vision projects isn't bad data or weak hardware. It's the endless comparison of architectures. I've watched teams burn months testing YOLO against Faster R-CNN against DETR while their actual problem sat unsolved. The fix is brutal simplicity: pick one model that fits your constraints, train it on a dataset that matches your deployment scene, and measure it with one metric. That's it. This walkthrough is for engineers and technical leads who need to get a vision system working, not win a benchmark.

1. Define the job before you touch a model

Computer vision covers everything from medical imaging to industrial inspection, but your project is one narrow task. Write it down as a single sentence: "Detect scratches on metal panels moving at 2 meters per second" or "Count people entering a store from a ceiling camera." If you can't write that sentence, you're not ready to pick a model.

Classification assigns one label to an image. Detection finds objects and draws boxes. Segmentation labels every pixel. These are different problems. Object detection requires both recognizing objects and localizing them with bounding boxes, unlike image classification which only assigns a label (PLOS ONE). Pick the wrong category and no amount of tuning will save you.

2. Choose one architecture that matches your latency and hardware

If you need real-time inference on a Jetson or a phone, YOLO is the obvious starting point. The original YOLO model processes images at 45 frames per second, and the smaller Fast YOLO hits 155 FPS (YOLO paper). That's not a marginal speed advantage; it's the difference between a live system and a slideshow. YOLOv4 later reached 43.5% AP on MS COCO at about 65 FPS on a Tesla V100 (YOLOv4 paper).

If you're running on a server and accuracy matters more than latency, Faster R-CNN with a VGG-16 backbone runs at 5 FPS and uses 300 proposals per image (Faster R-CNN paper). That's fine for offline inspection or batch processing. For mobile and edge devices, MobileNet uses depthwise separable convolutions for lightweight inference (PLOS ONE). Don't mix these up. A 5 FPS model on a drone is a crash. A 155 FPS model on a server is wasted capacity.

What can go wrong: you pick YOLO because it's fast, then discover your objects are tiny and clustered. YOLO makes more localization errors than two-stage detectors (YOLO paper). If your use case is counting pills on a tray, those localization errors will destroy your counts. In that case, switch to a two-stage detector or add a Feature Pyramid Network, which builds high-level semantic feature maps at all scales with marginal extra cost (Feature Pyramid Networks paper).

3. Match your dataset to your deployment scene

Training on ImageNet and deploying in a parking garage is a classic mistake. ImageNet indexes 14,197,122 images across 21,841 synsets (ImageNet official site), but it's a classification dataset. For detection, MS COCO has 2.5 million labeled instances in 328,000 images across 91 object types (Microsoft COCO paper). Those are general scenes. Your factory floor is not a general scene.

If you're building for autonomous driving, use KITTI or nuScenes. nuScenes carries 6 cameras, 5 radars, and 1 lidar with full 360-degree field of view across 1000 scenes (nuScenes dataset paper). That's the sensor suite you'll actually deploy. If you're doing medical segmentation, U-Net was designed for biomedical images and can be trained end-to-end from very few images (U-Net paper). Using a generic detector there is malpractice.

The practical move: collect 500 to 1,000 images from your actual camera, in your actual lighting, at your actual angle. Label them. Fine-tune a pre-trained model. A model pre-trained on ImageNet or COCO already knows edges, textures, and shapes. You're just teaching it your specific objects.

4. Measure with one metric and ship

Accuracy is useless for detection. Use Intersection over Union (IoU) and mean Average Precision (mAP). IoU tells you how well your predicted box overlaps the ground truth. mAP averages precision across recall levels. If your IoU threshold is 0.5 and your mAP is 0.75, you have a working detector. If it's 0.3, you have a science project.

Set a target before you train. For a warehouse robot detecting pallets, 0.7 mAP at 0.5 IoU is probably enough. For a medical device screening for diabetic retinopathy, the bar is higher. A deep learning system for diabetic retinopathy achieved 0.991 AUC on EyePACS-1 and 0.990 on Messidor-2 (JAMA 2016 deep learning diabetic retinopathy study). That's the level of validation required when a false negative can blind someone.

Once you hit your target, stop. Don't chase the last two points of mAP. Don't try Vision Transformer because it's newer. Ship the system, collect real-world failure cases, and retrain quarterly. The single best move is to pick one model, train it on your own data, and put it in front of users. Everything else is procrastination dressed up as engineering.

Sources

• PLOS ONE - https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0307461
• YOLO paper - https://arxiv.org/abs/1506.02640
• Faster R-CNN paper - https://arxiv.org/abs/1506.01497
• Microsoft COCO paper - https://arxiv.org/abs/1405.0312
• JAMA 2016 deep learning diabetic retinopathy study - https://pubmed.ncbi.nlm.nih.gov/27898976/