Stop Overfitting Your CV Demo: Pick the Right Detector for the Job

You Don't Need the Biggest Model

There's a common misconception in computer vision that the best-performing model on a benchmark like COCO is the right choice for every application. That's wrong. In practice, the best model is the one that fits your constraints: your data, your hardware, and your latency budget. I've seen teams burn months trying to deploy a state-of-the-art detector when a much simpler model would have solved the problem faster and more reliably.

This guide is for engineers and researchers who are building a computer vision system—maybe for medical imaging, autonomous driving, retail analytics, or something else entirely. We'll walk through the decision process step by step, from understanding your problem to evaluating your final model. By the end, you'll know how to choose a detector that actually works in the real world, not just on a leaderboard.

1. Define Your Task and Constraints

First, be precise about what you need. Are you classifying images, detecting objects, or segmenting pixels? For many applications, classification is enough, but if you need to localize objects, you're in detection territory. Object detection is fundamentally different from classification: you have to both recognize and localize with bounding boxes (PLOS ONE). That extra requirement changes everything about your model choice.

Next, list your constraints. What hardware will you run on? Is it a server with a powerful GPU, or a mobile phone? How fast does inference need to be? Real-time detection at 30 frames per second demands a different approach than offline processing of medical scans. And how much labeled data do you have? If you're starting from scratch, you might need a model that works well with transfer learning or even zero-shot.

For example, if you're building an autonomous vehicle system, you'll need to detect pedestrians and cars in 3D space, often using lidar and cameras (nuScenes dataset paper). That's a far more complex task than counting products on a shelf. But the principle holds: define the problem before you pick the model.

2. Match the Model to Your Data Scale

Your dataset size is a huge factor. If you have only a few thousand images, you're not going to train a massive Vision Transformer from scratch. You'll want to leverage a pre-trained model and fine-tune it. But which one?

For moderate-sized datasets, classic CNNs like ResNet are still workhorses. ResNet's residual learning made it possible to train very deep networks, and it won ILSVRC 2015 with an ensemble achieving 3.57% error on ImageNet (Deep Residual Learning paper). It's reliable and well-understood.

If you have a small dataset and need a lightweight model, MobileNetV2 is designed for mobile and edge devices, using depthwise separable convolutions to keep computation low (MobileNetV2 paper). But if you have a huge dataset and lots of compute, a Vision Transformer (ViT) might be worth it. ViTs apply a pure transformer to image patches and, when pre-trained on large data, can match or beat CNNs (Vision Transformer (ViT) paper).

But here's the thing: you don't always need the latest architecture. For many tasks, a simple CNN with good data augmentation is enough. The key is to match the model complexity to your data richness.

3. Speed vs. Accuracy: The Real Trade-off

In real-world applications, speed is often as important as accuracy. You might need to process video in real time, like in a surveillance system. That's where YOLO shines. YOLO frames detection as a regression problem and predicts bounding boxes and class probabilities in one pass (YOLO paper). The original YOLO ran at 45 FPS on a GPU, and a smaller version hit 155 FPS (YOLO paper). That's fast enough for many real-time applications.

On the other end, two-stage detectors like Faster R-CNN are more accurate but slower. Faster R-CNN introduced a Region Proposal Network that shares features with the detection network, but it still runs at only 5 FPS with VGG-16 (Faster R-CNN paper). If you're processing offline, that's fine. But for real-time, you'll need something faster.

Feature Pyramid Networks (FPN) can help by using a top-down architecture with lateral connections to build feature maps at multiple scales, improving detection of small objects without much extra cost (Feature Pyramid Networks paper). But again, it's about the trade-off.

My recommendation: if you need real-time, go with YOLOv4 or a similar single-stage detector. YOLOv4 achieves 43.5% AP on COCO at 65 FPS on a Tesla V100 (YOLOv4 paper). That's a solid balance. If you need maximum accuracy and have the compute, consider EfficientDet, which uses a compound scaling method to achieve state-of-the-art 55.1 AP on COCO (EfficientDet paper). But be prepared for the computational cost.

4. Don't Forget Your Domain: Medical and Face Recognition

Specialized domains have their own best practices. In medical imaging, for example, accuracy is paramount, but so is interpretability. The FDA has approved several AI/ML-based computer-aided detection devices, and the first autonomous AI diagnostic system, IDx-DR, was authorized to screen for diabetic retinopathy without a clinician interpreting the image (IDx-DR pivotal trial (npj Digital Medicine)). That was based on a pivotal trial of 900 patients. If you're building a medical tool, you'll need to meet regulatory standards and prove your model's performance on large, validated datasets.

For face recognition, NIST's Face Recognition Vendor Test (FRVT) is the gold standard. It measures accuracy and speed for one-to-many identification, with galleries of at least 10 million identities (NIST Face Recognition Vendor Test). The FRVT also documents demographic effects, which is crucial for fairness. If you're building a face recognition system, you should aim to pass FRVT-like evaluations, and consider using ArcFace, which adds an additive angular margin loss to improve discriminative power (ArcFace paper).

5. Evaluate with the Right Metrics and Datasets

Don't just look at accuracy. Use metrics like precision, recall, and Intersection over Union (IoU) to understand your model's strengths and weaknesses (Computer Vision courses (Southampton / NTNU)). For detection, mAP (mean Average Precision) is standard, but it can hide issues with small objects or rare classes.

Choose evaluation datasets that match your application. For autonomous driving, KITTI is a real-world benchmark with stereo, optical flow, and 3D object detection, with up to 15 cars and 30 pedestrians per image (KITTI Vision Benchmark Suite). For general object detection, COCO has 328,000 images with 2.5 million labeled instances across 91 object types (Microsoft COCO paper). But COCO is not the be-all and end-all. If you're working on urban scenes, Cityscapes provides dense pixel annotations for 30 classes across 50 cities (Cityscapes Dataset official site).

And remember, your model might behave differently in the real world than on a benchmark. The NIST FRVT found demographic differentials in face recognition accuracy across 18 million images of 8 million people (NIST Face Recognition Vendor Test). So test your model on diverse data that reflects your actual deployment scenario.

6. What Can Go Wrong: The Pitfalls

Here's the warning: you can easily overfit to a benchmark and end up with a model that fails in production. For example, if you train on ImageNet and deploy in a completely different domain, your model may not generalize. Even worse, adversarial examples—small perturbations that cause misclassification—can fool neural networks (Explaining and Harnessing Adversarial Examples paper). That's a security risk in applications like face recognition or autonomous driving.

Another pitfall is ignoring interpretability. If you're a doctor using a model to diagnose diabetic retinopathy, you need to know why it made a decision. Grad-CAM provides visual explanations by highlighting the regions in an image that influenced the model's decision (Grad-CAM paper). That's not just a nice-to-have; it's essential for building trust and debugging.

Finally, don't underestimate the importance of preprocessing and data quality. Even the best model will fail on noisy or poorly normalized images. Standard preprocessing like grayscale conversion, normalization, and noise reduction is still critical (Computer Vision courses (Southampton / NTNU)).

Sources

• PLOS ONE - https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0307461
• YOLO paper - https://arxiv.org/abs/1506.02640
• Faster R-CNN paper - https://arxiv.org/abs/1506.01497
• Feature Pyramid Networks paper - https://arxiv.org/abs/1612.03144
• EfficientDet paper - https://arxiv.org/abs/1911.09070
• NIST Face Recognition Vendor Test - https://www.nist.gov/programs-projects/face-recognition-vendor-test-frvt