Choosing a Real-Time Object Detector: YOLOv4 vs. Faster R-CNN

You're building a system that needs to detect objects in a live video stream, say for a retail analytics dashboard or a warehouse safety monitor. The question you type into a search engine is: “Which object detection model should I use for real-time?” It's a deceptively simple question, and the answer depends on whether you prioritize raw accuracy or the ability to run on modest hardware without melting it.

In this field report, we walk through a realistic scenario: you're a machine learning engineer at a startup that just got funding to build a proof-of-concept for detecting pallets and forklifts in warehouse security cameras. You have a single GPU (something like a Tesla V100), a tight deadline, and a boss who keeps saying “make it fast.” We'll look at the two main families of detectors—one-stage (YOLO) and two-stage (Faster R-CNN)—and see which one you should actually pick.

The Speed vs. Accuracy Trade-Off

Object detection is fundamentally about two things: recognizing what's in an image and localizing it with a bounding box. Classification alone won't cut it here. The classic divide is between one-stage detectors that do everything in a single pass and two-stage detectors that first propose regions and then classify them. YOLO—You Only Look Once—treats detection as a regression problem, predicting boxes and class probabilities directly from the full image (YOLO paper). Faster R-CNN, on the other hand, uses a Region Proposal Network to generate candidate boxes and then classifies each one (Faster R-CNN paper).

You know the drill: two-stage detectors tend to be more accurate, but they're slower. One-stage detectors are faster, but historically they lagged in accuracy. That gap has narrowed, but the choice still depends on your constraints.

What the Benchmarks Actually Say

Let's look at the numbers you'd care about. On the MS COCO dataset, YOLOv4 achieves 43.5% AP (average precision) at about 65 FPS on a Tesla V100 (YOLOv4 paper). That's a solid accuracy number for real-time. Faster R-CNN, with a VGG-16 backbone, runs at only 5 FPS on a GPU while using 300 proposals per image (Faster R-CNN paper). That's a 13x speed difference, but Faster R-CNN was state-of-the-art on PASCAL VOC and COCO in 2015 (Faster R-CNN paper). The accuracy gap is real but not enormous—Faster R-CNN's AP on COCO is not given in our fact base, but it's known to be lower than YOLOv4's, especially after YOLOv4's improvements.

But wait—there's a catch. Faster R-CNN's 5 FPS is with an older backbone. Modern two-stage detectors are faster, but the fact base doesn't give us a comparable number. So for the sake of this comparison, we're sticking to the facts we have: YOLOv4 is designed for real-time, and Faster R-CNN is not.

The Real-World Scenario: Warehouse Cameras

Imagine you're processing 10 cameras at 30 FPS each. That's 300 frames per second total. With YOLOv4 at 65 FPS on a single V100, you'd need about 5 GPUs just to keep up—and that's before considering the overhead of decoding video streams. With Faster R-CNN at 5 FPS, you'd need 60 GPUs, which is laughable. So for any real-time application, YOLO is the only practical choice if you care about latency.

But what about accuracy? In your warehouse, a missed forklift could be a safety hazard. YOLOv4's 43.5% AP means it's not perfect, but it's good enough for many tasks. Faster R-CNN might catch more true positives, but at the cost of missing the real-time requirement entirely. Your boss won't accept a system that processes video at 5 FPS.

Deployment Constraints: Edge Devices

Maybe you don't have a V100. Maybe you need to run on an edge device like a Jetson Nano or a smartphone. That's where MobileNet comes in—it uses depthwise separable convolutions for efficient inference (PLOS ONE). But MobileNet is a backbone, not a full detector. You could pair it with YOLO or SSD. The fact base mentions MobileNetV2 as a lightweight backbone, and it's designed for mobile and edge devices (MobileNetV2 paper). So if your deployment target is constrained hardware, you'd likely choose a YOLO variant with a MobileNet backbone.

However, the fact base doesn't give us specific FPS numbers for MobileNet-based detectors on edge devices. So we'll stick to the general principle: one-stage detectors are inherently faster, and YOLOv4 is a prime example.

Accuracy vs. Speed: A Table to Compare

Here's a quick comparison of the two detectors based on the facts we have:

The Hidden Costs: Beyond FPS

Speed isn't the only thing that matters. YOLOv4 combines a bunch of tricks—weighted residual connections, cross-stage partial connections, cross mini-batch normalization, self-adversarial training, Mish activation, Mosaic data augmentation, DropBlock regularization, and CIoU loss (YOLOv4 paper). That's a complex recipe that can be harder to implement and tune. Faster R-CNN is simpler conceptually, but it's slower and has its own complexities with region proposals.

Also, consider the ecosystem. YOLO has a huge community, and you can find pre-trained models easily. Faster R-CNN is also well-supported, but it's often used in research settings where speed isn't the priority.

A Word of Caution

Quick warning: Don't chase the highest AP number if your application can't run in real-time. A model that's 10% more accurate but 10x slower is useless for video processing.

What I'd Actually Do

For the warehouse scenario, I'd go with YOLOv4. It gives you the speed you need, and the accuracy is respectable. If you have a V100, you can process multiple streams with a single GPU. If you need even more speed, you could use a smaller variant like YOLOv4-tiny (which isn't in our fact base, but it's a common choice). The bottom line: when real-time is non-negotiable, YOLO is your friend. Faster R-CNN is best left for offline analysis where you can afford the wait.

But don't just take my word for it. Test both on your own data. Set up a quick experiment, measure the mAP and FPS on your specific hardware, and see which one meets your requirements. The facts we have are a starting point, but your mileage may vary.

Sources

• YOLO paper - https://arxiv.org/abs/1506.02640
• Faster R-CNN paper - https://arxiv.org/abs/1506.01497
• YOLOv4 paper - https://arxiv.org/abs/2004.10934
• PLOS ONE - https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0307461
• MobileNetV2 paper - https://arxiv.org/abs/1801.04381