diff --git a/README.md b/README.md index 8c1a36d..f0749de 100644 --- a/README.md +++ b/README.md @@ -518,12 +518,12 @@ LMDeploy abstracts the complex inference process of multi-modal Vision-Language #### A 'Hello, world' Example ```python -from lmdeploy import pipeline, TurbomindEngineConfig +from lmdeploy import pipeline, TurbomindEngineConfig, ChatTemplateConfig from lmdeploy.vl import load_image model = 'OpenGVLab/InternVL3-8B' image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg') -pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=16384, tp=1)) +pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=16384, tp=1), chat_template_config=ChatTemplateConfig(model_name='internvl2_5')) response = pipe(('describe this image', image)) print(response.text) ``` @@ -535,12 +535,12 @@ If `ImportError` occurs while executing this case, please install the required d When dealing with multiple images, you can put them all in one list. Keep in mind that multiple images will lead to a higher number of input tokens, and as a result, the size of the context window typically needs to be increased. ```python -from lmdeploy import pipeline, TurbomindEngineConfig +from lmdeploy import pipeline, TurbomindEngineConfig, ChatTemplateConfig from lmdeploy.vl import load_image from lmdeploy.vl.constants import IMAGE_TOKEN model = 'OpenGVLab/InternVL3-8B' -pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=16384, tp=1)) +pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=16384, tp=1), chat_template_config=ChatTemplateConfig(model_name='internvl2_5')) image_urls=[ 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg', @@ -558,11 +558,11 @@ print(response.text) Conducting inference with batch prompts is quite straightforward; just place them within a list structure: ```python -from lmdeploy import pipeline, TurbomindEngineConfig +from lmdeploy import pipeline, TurbomindEngineConfig, ChatTemplateConfig from lmdeploy.vl import load_image model = 'OpenGVLab/InternVL3-8B' -pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=16384, tp=1)) +pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=16384, tp=1), chat_template_config=ChatTemplateConfig(model_name='internvl2_5')) image_urls=[ "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg", @@ -578,11 +578,11 @@ print(response) There are two ways to do the multi-turn conversations with the pipeline. One is to construct messages according to the format of OpenAI and use above introduced method, the other is to use the `pipeline.chat` interface. ```python -from lmdeploy import pipeline, TurbomindEngineConfig, GenerationConfig +from lmdeploy import pipeline, TurbomindEngineConfig, GenerationConfig, ChatTemplateConfig from lmdeploy.vl import load_image model = 'OpenGVLab/InternVL3-8B' -pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=16384, tp=1)) +pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=16384, tp=1), chat_template_config=ChatTemplateConfig(model_name='internvl2_5')) image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg') gen_config = GenerationConfig(top_k=40, top_p=0.8, temperature=0.8) @@ -597,7 +597,7 @@ print(sess.response.text) LMDeploy's `api_server` enables models to be easily packed into services with a single command. The provided RESTful APIs are compatible with OpenAI's interfaces. Below are an example of service startup: ```shell -lmdeploy serve api_server OpenGVLab/InternVL3-8B --server-port 23333 --tp 1 +lmdeploy serve api_server OpenGVLab/InternVL3-8B --chat-template internvl2_5 --server-port 23333 --tp 1 ``` To use the OpenAI-style interface, you need to install OpenAI: