Why I Still Reach for Faster R-CNN, YOLO, and Mask R-CNN First

I was on a call last month with a team that had spent three months trying to get a DETR variant to converge on a custom dataset of 5,000 images. They were about to miss a product deadline. I asked if they'd tried YOLOv5. Silence. Two weeks later, they had a working prototype. That's not an isolated story.

Look, I get it. Transformers are cool. Attention mechanisms are elegant. But if you're building something that needs to work, not just impress reviewers, you might be overcomplicating your life.

Start with what's boring and reliable

Faster R-CNN, YOLO, and Mask R-CNN have been around for years. That's not a weakness. It means when something breaks, you can find the answer on Stack Overflow. It means the pretrained weights actually work. It means you can explain to your product manager why the model is slow without drawing a diagram of multi-head attention.

Let's talk numbers. Faster R-CNN with a VGG-16 backbone runs at about 5 fps on a modern GPU. On a Jetson Xavier NX, I've squeezed YOLOv4-tiny to 30 fps at 416x416 input—enough for real-time people counting. Mask R-CNN adds a mask branch for maybe 10-15% extra inference time. These aren't theoretical benchmarks; these are numbers from actual deployments I've been involved with.

And here's a concrete scenario: a client needed to detect defects on a conveyor belt. 15 frames per second, 4 classes, 2,000 annotated images. We started with a YOLOv4 model. It hit 0.89 mAP after two days of fine-tuning. Would a Swin Transformer have done better? Maybe 0.91. But it would have taken two weeks to get the training pipeline right, and the edge device would have choked. We shipped. That's what matters.

The accuracy argument (and why it's overblown)

Yes, newer models are more accurate on paper. EfficientDet-D7 gets 55.1 AP on COCO. Swin Transformer gets 58.7 box AP. DETR removes anchor boxes and NMS entirely. Impressive. But those numbers come from massive datasets and carefully tuned training recipes. On your 3,000-image dataset, the gap shrinks. Sometimes it disappears.

I've seen a project where a team spent six weeks tuning a DETR model to get 2% more mAP than their YOLOv5 baseline. Six weeks. For 2%. Meanwhile, the baseline was already in production, collecting real-world data that could have been used to improve it further.

Don't get me wrong—if you're competing in a benchmark or have a research budget, go for the latest. But if you're trying to ship a product, the marginal gain rarely justifies the engineering cost.

Tooling and community matter more than you think

Pretrained models for the classic trio are everywhere. TensorFlow's Object Detection API, PyTorch's torchvision, even OpenCV's DNN module. You can fine-tune on a few hundred images with transfer learning and get decent results. Newer models often need large-scale pretraining to shine. ViT, for example, only really works when pretrained on massive datasets like JFT-300M. MAE gets 87.8% top-1 on ImageNet, but that's with a ViT-Huge and masking 75% of patches. Not exactly plug-and-play.

Evaluation is another thing. With Faster R-CNN or YOLO, you compute IoU, precision, recall—standard stuff. Some newer models introduce custom loss functions or matching strategies that make debugging a nightmare. I once spent a week trying to figure out why a DETR model was predicting empty boxes. Turns out, the Hungarian matching was misconfigured. With YOLO, I would have caught that in an hour.

What I actually do

My default recipe: start with YOLOv5 or YOLOv8 for detection. If I need segmentation, Mask R-CNN. If I need high accuracy and have time, Faster R-CNN with a ResNet-50 FPN backbone. Only after I have a working pipeline and a clear understanding of failure cases do I consider upgrading. And even then, I ask: what specific problem am I solving? If it's small object detection, maybe I'll try a feature pyramid tweak. If it's crowded scenes, maybe I'll look at a transformer. But I don't start there.

One more thing: deployment. Getting a transformer to run efficiently on an edge device is still an adventure. ONNX export can be finicky. TensorRT support is improving but not universal. With YOLO, you have Darknet, OpenVINO, TensorRT, and more. It just works.

The bottom line

Choose the simplest model that meets your requirements. That's it. The classic detection trio isn't glamorous, but it's reliable. And in production, reliable beats fancy every time.

Sources

• Faster R-CNN paper - https://arxiv.org/abs/1506.01497
• YOLO paper - https://arxiv.org/abs/1506.02640
• Mask R-CNN paper - https://arxiv.org/abs/1703.06870
• YOLOv4 paper - https://arxiv.org/abs/2004.10934
• EfficientDet paper - https://arxiv.org/abs/1911.09070
• Swin Transformer paper - https://arxiv.org/abs/2103.14030