Upload README.md with huggingface_hub
This commit is contained in:
Weiyun Wang
system
parent
c2d2995e3d
commit
291ac355a1
32
README.md
32
README.md
@ -33,7 +33,7 @@ We introduce InternVL3, an advanced multimodal large language model (MLLM) serie
|
|||||||
Compared to InternVL 2.5, InternVL3 exhibits superior multimodal perception and reasoning capabilities, while further extending its multimodal capabilities to encompass tool usage, GUI agents, industrial image analysis, 3D vision perception, and more.
|
Compared to InternVL 2.5, InternVL3 exhibits superior multimodal perception and reasoning capabilities, while further extending its multimodal capabilities to encompass tool usage, GUI agents, industrial image analysis, 3D vision perception, and more.
|
||||||
Additionally, we compare InternVL3 with Qwen2.5 Chat models, whose corresponding pre-trained base models are employed as the initialization of the langauge component in InternVL3. Benefitting from Native Multimodal Pre-Training, the InternVL3 series achieves even better overall text performance than the Qwen2.5 series.
|
Additionally, we compare InternVL3 with Qwen2.5 Chat models, whose corresponding pre-trained base models are employed as the initialization of the langauge component in InternVL3. Benefitting from Native Multimodal Pre-Training, the InternVL3 series achieves even better overall text performance than the Qwen2.5 series.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## InternVL3 Family
|
## InternVL3 Family
|
||||||
|
|
||||||
@ -49,7 +49,7 @@ In the following table, we provide an overview of the InternVL3 series.
|
|||||||
| InternVL3-38B | [InternViT-6B-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5) | [Qwen2.5-32B](https://huggingface.co/Qwen/Qwen2.5-32B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL3-38B) |
|
| InternVL3-38B | [InternViT-6B-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5) | [Qwen2.5-32B](https://huggingface.co/Qwen/Qwen2.5-32B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL3-38B) |
|
||||||
| InternVL3-78B | [InternViT-6B-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5) | [Qwen2.5-72B](https://huggingface.co/Qwen/Qwen2.5-72B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL3-78B) |
|
| InternVL3-78B | [InternViT-6B-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5) | [Qwen2.5-72B](https://huggingface.co/Qwen/Qwen2.5-72B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL3-78B) |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## Model Architecture
|
## Model Architecture
|
||||||
|
|
||||||
@ -106,39 +106,39 @@ In this work, we use the Best-of-N evaluation strategy and employ [VisualPRM-8B]
|
|||||||
|
|
||||||
### Multimodal Reasoning and Mathematics
|
### Multimodal Reasoning and Mathematics
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### OCR, Chart, and Document Understanding
|
### OCR, Chart, and Document Understanding
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Multi-Image & Real-World Comprehension
|
### Multi-Image & Real-World Comprehension
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Comprehensive Multimodal & Hallucination Evaluation
|
### Comprehensive Multimodal & Hallucination Evaluation
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Visual Grounding
|
### Visual Grounding
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Multimodal Multilingual Understanding
|
### Multimodal Multilingual Understanding
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Video Understanding
|
### Video Understanding
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### GUI Grounding
|
### GUI Grounding
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Spatial Reasoning
|
### Spatial Reasoning
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## Evaluation on Language Capability
|
## Evaluation on Language Capability
|
||||||
|
|
||||||
@ -146,8 +146,7 @@ We compare InternVL3 with Qwen2.5 Chat models, whose corresponding pre-trained
|
|||||||
Benefitting from Native Multimodal Pre-Training, the InternVL3 series achieves even better overall text performance than the Qwen2.5 series.
|
Benefitting from Native Multimodal Pre-Training, the InternVL3 series achieves even better overall text performance than the Qwen2.5 series.
|
||||||
Please note that the evaluation scores of Qwen2.5 series may differ from those officially reported, as we have adopted the prompt versions provided in the table across all datasets for OpenCompass evaluation.
|
Please note that the evaluation scores of Qwen2.5 series may differ from those officially reported, as we have adopted the prompt versions provided in the table across all datasets for OpenCompass evaluation.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## Ablation Study
|
## Ablation Study
|
||||||
|
|
||||||
@ -157,20 +156,19 @@ We conduct experiments on the InternVL2-8B model while keeping its architecture,
|
|||||||
|
|
||||||
The evaluation results in the Figure below shows that the model with native multimodal pre-training exhibits performance on most benchmarks that is comparable to the fully multi-stage-trained InternVL2-8B baseline. Furthermore, when followed by instruction tuning on higher-quality data, the model demonstrates further performance gains across evaluated multimodal tasks. These findings underscore the efficiency of native multimodal pre-training in imparting powerful multimodal capabilities to MLLMs.
|
The evaluation results in the Figure below shows that the model with native multimodal pre-training exhibits performance on most benchmarks that is comparable to the fully multi-stage-trained InternVL2-8B baseline. Furthermore, when followed by instruction tuning on higher-quality data, the model demonstrates further performance gains across evaluated multimodal tasks. These findings underscore the efficiency of native multimodal pre-training in imparting powerful multimodal capabilities to MLLMs.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Mixed Preference Optimization
|
### Mixed Preference Optimization
|
||||||
|
|
||||||
As shown in the table below, models fine-tuned with MPO demonstrate superior reasoning performance across seven multimodal reasoning benchmarks compared to their counterparts without MPO. Specifically, InternVL3-78B and InternVL3-38B outperform their counterparts by 4.1 and 4.5 points, respectively. Notably, the training data used for MPO is a subset of that used for SFT, indicating that the performance improvements primarily stem from the training algorithm rather than the training data.
|
As shown in the table below, models fine-tuned with MPO demonstrate superior reasoning performance across seven multimodal reasoning benchmarks compared to their counterparts without MPO. Specifically, InternVL3-78B and InternVL3-38B outperform their counterparts by 4.1 and 4.5 points, respectively. Notably, the training data used for MPO is a subset of that used for SFT, indicating that the performance improvements primarily stem from the training algorithm rather than the training data.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Variable Visual Position Encoding
|
### Variable Visual Position Encoding
|
||||||
|
|
||||||
As reported in the table below, the introduction of V2PE leads to significant performance gains across most evaluation metrics. In addition, our ablation studies—by varying the positional increment \\( \delta \\)—reveal that even for tasks primarily involving conventional contexts, relatively small \\( \delta \\) values can achieve optimal performance. These findings provide important insights for future efforts aimed at refining position encoding strategies for visual tokens in MLLMs.
|
As reported in the table below, the introduction of V2PE leads to significant performance gains across most evaluation metrics. In addition, our ablation studies—by varying the positional increment \\( \delta \\)—reveal that even for tasks primarily involving conventional contexts, relatively small \\( \delta \\) values can achieve optimal performance. These findings provide important insights for future efforts aimed at refining position encoding strategies for visual tokens in MLLMs.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user