Stop Tuning Hyperparameters: Train on SimCLR First

Who This Is For

You're a computer vision developer who's tired of babysitting annotation tools. You've got a pile of unlabeled images—maybe 10,000 product shots, medical scans, or traffic camera frames—and you need a model that actually works. Common advice says: label everything, then train a CNN. That's wrong. You should train a self-supervised model first, specifically SimCLR. The paper (SimCLR paper) shows that a linear classifier trained on SimCLR representations hits 76.5% top-1 accuracy on ImageNet—matching a supervised ResNet-50. And when you fine-tune on just 1% of the labels, you get 85.8% top-5 accuracy, beating AlexNet with 100x fewer labels. That's not a trick; it's a smarter way to use your data.

Step 1: Ditch the Hand-Crafted Features

Forget SIFT, HOG, LBP, and ORB (Computer Vision courses). They're useful for teaching, but in practice, they're obsolete. CNNs learn features hierarchically (Computer Vision courses), and self-supervised pretraining learns even more general features. The old way: extract features, then train a classifier. The new way: let the model learn features from the data itself, without any labels. You'll thank me later.

Step 2: Set Up SimCLR in Your Pipeline

SimCLR is a contrastive learning framework. It works by pulling together augmented views of the same image and pushing apart views of different images. The key is data augmentation—the paper (SimCLR paper) says composition of augmentations is critical. Here's the concrete recipe:

1. Start with your unlabeled dataset. Any collection of images works. You don't need labels, but you need enough variety.
2. Apply random crops, color distortions, and Gaussian blur. These are the standard augmentations. The SimCLR paper (SimCLR paper) uses them, and they're the reason the model learns useful representations.
3. Use a standard ResNet or EfficientNet backbone. The original SimCLR uses a ResNet-50, but you can swap in anything. EfficientNet scales nicely (EfficientNet paper), but keep it simple first.
4. Train with a contrastive loss. The model learns to maximize agreement between augmented views. No labels needed.
5. Once trained, take the encoder and add a linear classifier on top. Train that on your labeled data (even a tiny amount).

That's it. You've just built a model that outperforms a supervised baseline on 1% of the labels (SimCLR paper).

What can go wrong: If you don't tune the augmentation strength, your representations will be garbage. Too weak, and the model cheats by recognizing trivial similarities. Too strong, and the model can't learn anything. The paper (SimCLR paper) emphasizes this, so test a few settings.

Step 3: Compare with Supervised Baselines

Don't take my word for it. Run a quick experiment. Train a supervised ResNet-50 on your labeled data (the usual way), and train a SimCLR model the way I described. Then fine-tune the SimCLR encoder on the same labels. Compare accuracy on a validation set. The SimCLR paper (SimCLR paper) reports that its representations match a supervised ResNet-50 on ImageNet when using a linear classifier. You'll see the same pattern on your data, especially if you have few labels.

Numbers from SimCLR paper (SimCLR paper). You can replicate this with your own data.

Step 4: Scale Up with More Unlabeled Data

SimCLR shines when you have a lot of unlabeled data. The more you have, the better the representations. That's because contrastive learning needs variety to learn discriminative features. If you have 100,000 unlabeled images, use them. If you have a million, even better. The paper (SimCLR paper) used 1.28M images for pretraining, but you don't need that many to see a benefit. Even a few thousand helps.

For comparison, ImageNet has 1.43M annotated images (ILSVRC2012, from ImageNet Large Scale Visual Recognition Challenge (IJCV 2015)), but you don't need that much. SimCLR shows you can leverage unlabeled data that's easier to collect.

What About Other Self-Supervised Methods?

SimCLR is a great starting point, but it's not the only option. Masked autoencoders (MAE) are another powerful choice—they mask random patches and reconstruct the missing pixels, achieving 87.8% top-1 accuracy on ImageNet with only ImageNet-1K data (MAE paper). CLIP learns from image-text pairs and can zero-shot transfer (CLIP paper). For a tutorial, SimCLR is simpler to implement and understand. Start there, then experiment.

Final Takeaway

Stop labeling everything. Train a SimCLR model on your unlabeled data first. It's a proven way to get state-of-the-art representations without the annotation bottleneck. The numbers don't lie: SimCLR matches supervised performance on ImageNet with zero labels (SimCLR paper). You'll save weeks of labeling time and still get a model that works. Try it on your next project.

Sources

• SimCLR paper - https://arxiv.org/abs/2002.05709
• EfficientNet paper - https://arxiv.org/abs/1905.11946
• Computer Vision courses (Southampton / NTNU) - https://www.ntnu.edu/studies/courses/TDT4265
• ImageNet Large Scale Visual Recognition Challenge (IJCV 2015) - https://arxiv.org/abs/1409.0575