Compare commits
No commits in common. "6ee8a6d6f44b191d9ed1516f1d83636623862433" and "c2d2995e3d9d3f90fc9967f239b42f344016b38b" have entirely different histories.
6ee8a6d6f4
...
c2d2995e3d
55
README.md
55
README.md
@ -1,24 +1,25 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
license: other
|
||||
license_name: qwen
|
||||
license_link: https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/main/LICENSE
|
||||
pipeline_tag: image-text-to-text
|
||||
library_name: transformers
|
||||
base_model:
|
||||
- OpenGVLab/InternVL3-8B-Instruct
|
||||
base_model_relation: finetune
|
||||
- OpenGVLab/InternViT-300M-448px-V2_5
|
||||
- Qwen/Qwen2.5-7B
|
||||
base_model_relation: merge
|
||||
datasets:
|
||||
- OpenGVLab/MMPR-v1.2
|
||||
- OpenGVLab/MMPR-v1.2
|
||||
language:
|
||||
- multilingual
|
||||
- multilingual
|
||||
tags:
|
||||
- internvl
|
||||
- custom_code
|
||||
- internvl
|
||||
- custom_code
|
||||
---
|
||||
|
||||
# InternVL3-8B
|
||||
|
||||
[\[📂 GitHub\]](https://github.com/OpenGVLab/InternVL) [\[📜 InternVL 1.0\]](https://huggingface.co/papers/2312.14238) [\[📜 InternVL 1.5\]](https://huggingface.co/papers/2404.16821) [\[📜 InternVL 2.5\]](https://huggingface.co/papers/2412.05271) [\[📜 InternVL2.5-MPO\]](https://huggingface.co/papers/2411.10442) [\[📜 InternVL3\]](https://huggingface.co/papers/2504.10479)
|
||||
[\[📂 GitHub\]](https://github.com/OpenGVLab/InternVL) [\[📜 InternVL 1.0\]](https://huggingface.co/papers/2312.14238) [\[📜 InternVL 1.5\]](https://huggingface.co/papers/2404.16821) [\[📜 InternVL 2.5\]](https://huggingface.co/papers/2412.05271) [\[📜 InternVL2.5-MPO\]](https://huggingface.co/papers/2411.10442) [\[📜 InternVL3\]](TBD)
|
||||
|
||||
[\[🆕 Blog\]](https://internvl.github.io/blog/) [\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/) [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL) [\[🚀 Quick Start\]](#quick-start) [\[📖 Documents\]](https://internvl.readthedocs.io/en/latest/)
|
||||
|
||||
@ -32,7 +33,7 @@ We introduce InternVL3, an advanced multimodal large language model (MLLM) serie
|
||||
Compared to InternVL 2.5, InternVL3 exhibits superior multimodal perception and reasoning capabilities, while further extending its multimodal capabilities to encompass tool usage, GUI agents, industrial image analysis, 3D vision perception, and more.
|
||||
Additionally, we compare InternVL3 with Qwen2.5 Chat models, whose corresponding pre-trained base models are employed as the initialization of the langauge component in InternVL3. Benefitting from Native Multimodal Pre-Training, the InternVL3 series achieves even better overall text performance than the Qwen2.5 series.
|
||||
|
||||
|
||||
|
||||
|
||||
## InternVL3 Family
|
||||
|
||||
@ -48,7 +49,7 @@ In the following table, we provide an overview of the InternVL3 series.
|
||||
| InternVL3-38B | [InternViT-6B-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5) | [Qwen2.5-32B](https://huggingface.co/Qwen/Qwen2.5-32B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL3-38B) |
|
||||
| InternVL3-78B | [InternViT-6B-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5) | [Qwen2.5-72B](https://huggingface.co/Qwen/Qwen2.5-72B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL3-78B) |
|
||||
|
||||
|
||||
|
||||
|
||||
## Model Architecture
|
||||
|
||||
@ -65,9 +66,9 @@ Notably, in InternVL3, we integrate the [Variable Visual Position Encoding (V2PE
|
||||
|
||||
### Native Multimodal Pre-Training
|
||||
|
||||
We propose a [Native Multimodal Pre-Training](https://huggingface.co/papers/2504.10479) approach that consolidates language and vision learning into a single pre-training stage.
|
||||
We propose a [Native Multimodal Pre-Training](TBD) approach that consolidates language and vision learning into a single pre-training stage.
|
||||
In contrast to standard paradigms that first train a language-only model and subsequently adapt it to handle additional modalities, our method interleaves multimodal data (e.g., image-text, video-text, or image-text interleaved sequences) with large-scale textual corpora. This unified training scheme allows the model to learn both linguistic and multimodal representations simultaneously, ultimately enhancing its capability to handle vision-language tasks without the need for separate alignment or bridging modules.
|
||||
Please see [our paper](https://huggingface.co/papers/2504.10479) for more details.
|
||||
Please see [our paper](TBD) for more details.
|
||||
|
||||
### Supervised Fine-Tuning
|
||||
|
||||
@ -105,39 +106,39 @@ In this work, we use the Best-of-N evaluation strategy and employ [VisualPRM-8B]
|
||||
|
||||
### Multimodal Reasoning and Mathematics
|
||||
|
||||
|
||||
|
||||
|
||||
### OCR, Chart, and Document Understanding
|
||||
|
||||
|
||||
|
||||
|
||||
### Multi-Image & Real-World Comprehension
|
||||
|
||||
|
||||
|
||||
|
||||
### Comprehensive Multimodal & Hallucination Evaluation
|
||||
|
||||
|
||||
|
||||
|
||||
### Visual Grounding
|
||||
|
||||
|
||||
|
||||
|
||||
### Multimodal Multilingual Understanding
|
||||
|
||||
|
||||
|
||||
|
||||
### Video Understanding
|
||||
|
||||
|
||||
|
||||
|
||||
### GUI Grounding
|
||||
|
||||
|
||||
|
||||
|
||||
### Spatial Reasoning
|
||||
|
||||
|
||||
|
||||
|
||||
## Evaluation on Language Capability
|
||||
|
||||
@ -145,7 +146,8 @@ We compare InternVL3 with Qwen2.5 Chat models, whose corresponding pre-trained
|
||||
Benefitting from Native Multimodal Pre-Training, the InternVL3 series achieves even better overall text performance than the Qwen2.5 series.
|
||||
Please note that the evaluation scores of Qwen2.5 series may differ from those officially reported, as we have adopted the prompt versions provided in the table across all datasets for OpenCompass evaluation.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Ablation Study
|
||||
|
||||
@ -155,19 +157,20 @@ We conduct experiments on the InternVL2-8B model while keeping its architecture,
|
||||
|
||||
The evaluation results in the Figure below shows that the model with native multimodal pre-training exhibits performance on most benchmarks that is comparable to the fully multi-stage-trained InternVL2-8B baseline. Furthermore, when followed by instruction tuning on higher-quality data, the model demonstrates further performance gains across evaluated multimodal tasks. These findings underscore the efficiency of native multimodal pre-training in imparting powerful multimodal capabilities to MLLMs.
|
||||
|
||||
|
||||
|
||||
|
||||
### Mixed Preference Optimization
|
||||
|
||||
As shown in the table below, models fine-tuned with MPO demonstrate superior reasoning performance across seven multimodal reasoning benchmarks compared to their counterparts without MPO. Specifically, InternVL3-78B and InternVL3-38B outperform their counterparts by 4.1 and 4.5 points, respectively. Notably, the training data used for MPO is a subset of that used for SFT, indicating that the performance improvements primarily stem from the training algorithm rather than the training data.
|
||||
|
||||
|
||||
|
||||
|
||||
### Variable Visual Position Encoding
|
||||
|
||||
As reported in the table below, the introduction of V2PE leads to significant performance gains across most evaluation metrics. In addition, our ablation studies—by varying the positional increment \\( \delta \\)—reveal that even for tasks primarily involving conventional contexts, relatively small \\( \delta \\) values can achieve optimal performance. These findings provide important insights for future efforts aimed at refining position encoding strategies for visual tokens in MLLMs.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Quick Start
|
||||
|
||||
@ -668,7 +671,7 @@ print(response)
|
||||
|
||||
## License
|
||||
|
||||
This project is released under the MIT License. This project uses the pre-trained Qwen2.5 as a component, which is licensed under the Apache-2.0 License.
|
||||
This project is released under the MIT License. This project uses the pre-trained Qwen2.5 as a component, which is licensed under the Qwen License.
|
||||
|
||||
## Citation
|
||||
|
||||
|
||||
@ -1,123 +0,0 @@
|
||||
# InternVL3-8B 在 Material Analyzer 中的使用说明
|
||||
|
||||
本目录存放 `InternVL3-8B` 模型文件。它是 Material Analyzer 的外部模型依赖，不应该提交到 `Material Analyzer` 项目仓库。
|
||||
|
||||
## 主要功能
|
||||
|
||||
`InternVL3-8B` 是视觉语言模型。在本项目中，它用于查看地板或瓷砖图片，并结合产品元数据生成结构化语义结果。
|
||||
|
||||
本项目使用它完成以下字段：
|
||||
|
||||
```json
|
||||
{
|
||||
"description": "",
|
||||
"material_type": "",
|
||||
"surface_finish": "",
|
||||
"grain_type": "",
|
||||
"color_family": "",
|
||||
"visual_style": "",
|
||||
"variation": "",
|
||||
"gloss_level": "",
|
||||
"tags": [],
|
||||
"confidence": 0.0,
|
||||
"field_confidence": {}
|
||||
}
|
||||
```
|
||||
|
||||
这些结果会写入每个资产目录中的：
|
||||
|
||||
```text
|
||||
semantic.json
|
||||
material.json
|
||||
```
|
||||
|
||||
## 本项目没有使用的能力
|
||||
|
||||
当前 Material Analyzer 只使用 InternVL3 做语义分析，不使用它生成图像 embedding。
|
||||
|
||||
图像 embedding 仍由以下 provider 负责：
|
||||
|
||||
- `local`：本地确定性颜色/纹理指纹
|
||||
- `http`：可选 DINOv2/CLIP 模型服务
|
||||
|
||||
## 推荐仓库管理方式
|
||||
|
||||
建议将本目录作为独立模型仓库或 Hugging Face 模型仓库管理。不要把模型权重放进业务代码仓库。
|
||||
|
||||
如果使用 Git 管理该模型目录，需要启用 Git LFS，并跟踪大文件：
|
||||
|
||||
```bash
|
||||
git lfs install
|
||||
git lfs track "*.safetensors"
|
||||
git lfs track "*.bin"
|
||||
git add .gitattributes
|
||||
```
|
||||
|
||||
## 本地放置路径
|
||||
|
||||
Material Analyzer 默认读取：
|
||||
|
||||
```text
|
||||
D:\go-demo\data\InternVL3-8B
|
||||
```
|
||||
|
||||
如果模型放在其他位置，需要在启动语义服务前设置：
|
||||
|
||||
```bash
|
||||
set INTERNVL3_MODEL_PATH=D:\path\to\InternVL3-8B
|
||||
```
|
||||
|
||||
## 启动语义服务
|
||||
|
||||
在 Material Analyzer 项目中运行：
|
||||
|
||||
```bash
|
||||
cd D:\go-demo\Material Analyzer\tools\vision_server
|
||||
python -m pip install -r requirements.txt
|
||||
set INTERNVL3_MODEL_PATH=D:\go-demo\data\InternVL3-8B
|
||||
python server.py
|
||||
```
|
||||
|
||||
服务默认监听：
|
||||
|
||||
```text
|
||||
http://127.0.0.1:5300/semantic
|
||||
```
|
||||
|
||||
## 在 Material Analyzer 中调用
|
||||
|
||||
回到 Material Analyzer 项目根目录：
|
||||
|
||||
```bash
|
||||
cd D:\go-demo\Material Analyzer
|
||||
go run . -sku I966106LP -semantic-provider internvl3 -semantic-url http://127.0.0.1:5300
|
||||
```
|
||||
|
||||
也可以和本地规则 provider 组合：
|
||||
|
||||
```bash
|
||||
go run . -semantic-provider local,internvl3 -semantic-url http://127.0.0.1:5300
|
||||
```
|
||||
|
||||
组合模式会做字段级融合，并根据置信度决定最终语义字段。
|
||||
|
||||
## 常用环境变量
|
||||
|
||||
```text
|
||||
INTERNVL3_MODEL_PATH 模型目录，默认 D:\go-demo\data\InternVL3-8B
|
||||
INTERNVL3_PORT 服务端口，默认 5300
|
||||
INTERNVL3_IMAGE_SIZE 输入图块尺寸，默认 448
|
||||
INTERNVL3_MAX_TILES 最大图块数，默认 4
|
||||
INTERNVL3_MAX_NEW_TOKENS 最大生成 token 数，默认 512
|
||||
INTERNVL3_TEMPERATURE 生成温度，默认 0.0
|
||||
INTERNVL3_LOAD_IN_8BIT 是否 8bit 量化加载
|
||||
INTERNVL3_LOAD_IN_4BIT 是否 4bit 量化加载
|
||||
INTERNVL3_USE_FLASH_ATTN 是否启用 flash attention
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- 模型权重体积很大，普通 Git 不适合直接管理。
|
||||
- 首次启动会加载多个 `.safetensors` 分片，耗时和显存占用都较高。
|
||||
- 如果显存不足，可以尝试 `INTERNVL3_LOAD_IN_8BIT=true` 或 `INTERNVL3_LOAD_IN_4BIT=true`，但需要环境支持 bitsandbytes。
|
||||
- 本项目语义服务接口只要求 `/semantic` 可用；业务代码不直接读取模型目录。
|
||||
@ -10,7 +10,7 @@ import torch
|
||||
import torch.nn.functional as F
|
||||
import torch.utils.checkpoint
|
||||
from einops import rearrange
|
||||
from timm.layers import DropPath
|
||||
from timm.models.layers import DropPath
|
||||
from torch import nn
|
||||
from transformers.activations import ACT2FN
|
||||
from transformers.modeling_outputs import (BaseModelOutput,
|
||||
|
||||
@ -123,8 +123,8 @@ class InternVLChatModel(PreTrainedModel):
|
||||
vit_embeds = vit_embeds.reshape(-1, C)
|
||||
print(f'warning: {e}, input_embeds[selected].shape={input_embeds[selected].shape}, '
|
||||
f'vit_embeds.shape={vit_embeds.shape}')
|
||||
n_token = min(selected.sum(), vit_embeds.size(0))
|
||||
input_embeds[selected][:n_token] = input_embeds[selected][:n_token] * 0.0 + vit_embeds[:n_token]
|
||||
n_token = selected.sum()
|
||||
input_embeds[selected] = input_embeds[selected] * 0.0 + vit_embeds[:n_token]
|
||||
|
||||
input_embeds = input_embeds.reshape(B, N, C)
|
||||
|
||||
|
||||
@ -268,7 +268,7 @@
|
||||
"<|video_pad|>"
|
||||
],
|
||||
"bos_token": null,
|
||||
"chat_template": "{%- if messages[0]['role'] == 'system' %}{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}{%- else %}{{- '<|im_start|>system\n\u4f60\u662f\u4e66\u751f\u00b7\u4e07\u8c61\uff0c\u82f1\u6587\u540d\u662fInternVL\uff0c\u662f\u7531\u4e0a\u6d77\u4eba\u5de5\u667a\u80fd\u5b9e\u9a8c\u5ba4\u3001\u6e05\u534e\u5927\u5b66\u53ca\u591a\u5bb6\u5408\u4f5c\u5355\u4f4d\u8054\u5408\u5f00\u53d1\u7684\u591a\u6a21\u6001\u5927\u8bed\u8a00\u6a21\u578b\u3002<|im_end|>\n' }}{%- endif %}{% for message in messages %}{%- if messages[0]['role'] != 'system' or not loop.first %}{{'<|im_start|>' + message['role'] + '\n'}}{% if message['content'] is string %}{{ message['content'] }}{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' %}{{ '<image>\n' }}{% elif content['type'] == 'video' %}{{ '<video>\n' }}{% elif content['type'] == 'text' %}{{ content['text'] }}{% endif %}{% endfor %}{% endif %}{{'<|im_end|>\n'}}{%- endif %}{% endfor %}{% if add_generation_prompt %}{{'<|im_start|>assistant\n' }}{% endif %}",
|
||||
"chat_template": "{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0]['role'] == 'system' %}\n {{- messages[0]['content'] }}\n {%- else %}\n {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n {%- endif %}\n {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n {%- if messages[0]['role'] == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n {%- else %}\n {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {{- '<|im_start|>' + message.role }}\n {%- if message.content %}\n {{- '\\n' + message.content }}\n {%- endif %}\n {%- for tool_call in message.tool_calls %}\n {%- if tool_call.function is defined %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n<tool_call>\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- '}\\n</tool_call>' }}\n {%- endfor %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n<tool_response>\\n' }}\n {{- message.content }}\n {{- '\\n</tool_response>' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|im_end|>",
|
||||
"errors": "replace",
|
||||
|
||||
Loading…
Reference in New Issue
Block a user