Multimodal Large Language Models for General-Purpose Vision-Language Intelligence: Architectures, Instruction Tuning, and Emergent Capabilities

Abstract

The convergence of large language modeling and vision-language pretraining led to rapid development of multimodal large language models during 2023. Earlier vision-language systems were typically optimized for predefined tasks such as captioning, retrieval, visual question answering, or recognition. Multimodal foundation models instead sought to connect pretrained visual encoders with large language models capable of instruction following and open-ended generation. This review examines the architectural evolution and principal capabilities of multimodal large language models available through 2023. Contrastive Language-Image Pretraining established transferable image-text alignment at scale, while Flamingo demonstrated few-shot multimodal learning by connecting frozen vision and language components. BLIP and BLIP-2 advanced bootstrapped vision-language pretraining, with BLIP-2 employing a lightweight Querying Transformer to bridge frozen image encoders and frozen large language models. LLaVA demonstrated visual instruction tuning using language-model-generated multimodal instruction data, while InstructBLIP systematically extended instruction tuning across diverse vision-language tasks. Concurrent systems including MiniGPT-4, Kosmos-1, PaLI, PaLI-X, and ImageBind explored different combinations of multimodal representation, language generation, and cross-modal transfer. The review analyzes visual encoders, connector modules, cross-attention, instruction tuning, frozen versus jointly trained components, multimodal benchmarks, and computational efficiency. Challenges identified in 2023 included visual hallucination, insufficient spatial reasoning, OCR limitations, weak fine-grained grounding, dataset contamination, safety, evaluation inconsistencies, and the difficulty of distinguishing genuine multimodal reasoning from language-prior exploitation.