Personalized Multimodal Understanding with RC-MLLM
RC-MLLM model is developed based on the Qwen2-VL model through a novel method called RCVIT (Region-level Context-aware Visual Instruction Tuning), using the specially constructed RCMU dataset for training. Its core feature is the capability for Region-level Context-aware Multimodal Understanding (RCMU). This means it can simultaneously understand both the visual content of specific regions/objects within an image and their associated textual information (utilizing bounding boxes coordinates), allowing it to respond to user instructions in a more context-aware manner. Simply put, RC-MLLM not only understands images but can also integrate the textual information linked to specific objects within the image for understanding. It achieves outstanding performance on RCMU tasks and is suitable for applications like personalized conversation.
📌 First build a multimodal personalized knowledge base, then perform personalized multimodal understanding with RC-MLLM
1. Build Multimodal Personalized Knowledge Base
📖 Upload images, click on people or objects in the images and fill in their personalized information, then save them to create a multimodal personalized knowledge base
Click on people or objects in the image to get a mask
| Object Image or Face Image (Select the type of image to upload) | Current Image | Upload Images | Input Personalized Information |
|---|
2. Personalized Multimodal Understanding with RC-MLLM
📖 Upload images and use the RC-MLLM model for personalized Q&A
| Input Image | Question |
|---|
✅ RC-MLLM model loaded successfully