Image Captioning with Frozen Self-Supervised Vision Encoders

Self-supervised vision models such as DINOv3 produce rich, spatially detailed patch-level features without any text supervision. These features excel on dense prediction tasks (segmen- tation, depth estimation, keypoint matching), yet they remain disconnected from the language modality. This thesis investigates how a frozen DINOv3 ViT-S/16 encoder can contribute to image captioning when used alone and when fused with a pretrained language-aligned encoder. The preliminary investigation documents three attempts to bridge DINOv3 to a frozen large language model (Qwen2.5-3B) using only DINOv3 as the vision backbone: a BLIP-2-style Q-Former with ITC, ITM, and ITG objectives; a LLaVA-style MLP projection into the LLM embedding space; and a dino.txt-style contrastive vision block pretraining stage with a frozen text encoder. All three produce captioning metrics at least an order of magnitude below pub- lished baselines. The BLIP-2 bridge generates repetitive, degenerate captions. The LLaVA pipeline produces fluent but entirely hallucinated descriptions. A post-hoc analysis uncovers a 100× L2 norm mismatch between projected visual tokens and native LLM text embeddings, di- agnosing the LLaVA failure mechanistically. The contrastive vision block pretraining underfits at our data scale. These failures motivate a pivot to dual-backbone fusion. We pair frozen DINOv3 with frozen SigLIP 2 (ViT-L/16) and route their patch tokens through a small learnable fusion mod- ule whose output feeds a compact causal decoder initialized from Qwen3-0.6B token embed- dings. Four fusion variants are compared: NoFusion (single-backbone baseline), ConcatFu- sion, SequentialEnrichment, and a bidirectional ParallelTextConditioned module with a learned sigmoid gate. On the COCO Karpathy validation split, SigLIP 2 alone reaches CIDEr 0.396 at 10,000 steps while DINOv3 alone reaches only 0.065, quantifying the cost of the miss- ing language alignment. The ParallelTC fusion matches SigLIP 2-only at matched steps and pulls ahead at 40,000 steps (CIDEr 0.456 vs. 0.441 at 30,000 SigLIP 2-only steps). On out- of-domain radiology (RoCov2), fusion does not improve over SigLIP 2 alone, consistent with both backbones being trained on natural images. This thesis contributes a systematic experimental account of the DINOv3-to-language modal- ity gap, a mechanistic diagnosis of the embedding scale mismatch, and a compact fusion recipe in which DINOv3 contributes fine-grained spatial structure while SigLIP 2 contributes pre- trained language alignment. The findings clarify when self-supervised features add captioning signal and when they only add parameters.