Self-Supervised Visual Representation Learning: Contrastive, Siamese, Clustering, and Self-Distillation Approaches

Abstract

The dependence of supervised deep learning on large quantities of manually annotated data represents an important constraint in computer vision. Self-supervised learning addresses this limitation by constructing supervisory signals directly from unlabeled observations and learning transferable representations before downstream fine-tuning. Between 2018 and 2021, self-supervised visual learning progressed from handcrafted pretext tasks toward contrastive learning, momentum-based encoders, clustering, Siamese architectures, redundancy reduction, and self-distillation. This review examines the principal approaches available through 2021. Contrastive predictive coding and instance-discrimination techniques established the foundations for learning representations through similarity relationships. Momentum Contrast introduced dynamic dictionaries and momentum encoders, while SimCLR demonstrated the importance of data augmentation, projection heads, and large training batches. Bootstrap Your Own Latent showed that competitive representations could be obtained without explicit negative samples. SwAV combined clustering with swapped prediction between augmented views, whereas SimSiam demonstrated the importance of stop-gradient operations within comparatively simple Siamese networks. Barlow Twins approached representation learning through redundancy reduction, while DINO demonstrated strong emergent properties when self-distillation was combined with Vision Transformers. The review compares these approaches in terms of negative samples, architectural asymmetry, augmentation strategy, memory requirements, representation collapse, and transferability. By the end of 2021, self-supervised learning had become a major alternative to conventional supervised pretraining and provided an increasingly practical route for exploiting large unlabeled image collections.