Minigpt-4

Enhance visual understanding with advanced large language models. We are currently preparing a lighter model that can run on a single 3090 GPU, which you will be able to run on your own machine. Please visit our GitHub page to stay updated. Minigpt-4 uses only one projection layer to align the frozen visual encoder from Blip-2 with the frozen LLM Vicuna. We use two stages to train Minigpt-4. The first conventional preprocessing stage using 4 A100s trains with approximately 5 million aligned image-text pairs. After the first stage, Vicuna is able to understand images. However, Vicuna's generative capabilities are severely impacted. To address this and improve usability, we propose a novel way to create high-quality image-text pairs through the model itself and chat with them. Based on this, we created a small (a total of 3500 pairs) but high-quality dataset. The second finetuning stage is trained on this dataset in a conversational template to significantly improve its generative reliability and overall usability. To our surprise, this stage is computationally efficient, requiring only about 7 minutes on a single A100. Many of the emerging visual-language capabilities produced by Minigpt-4 are similar to those demonstrated in GPT-4.

Minigpt-4 enhances visual-language understanding using advanced large language models. By aligning a frozen visual encoder with the frozen large language model Vicuna, it achieves GPT-4-like multimodal capabilities, such as generating detailed image descriptions and creating websites from handwritten drafts.