ms-swift+llamacpp+ollama微调部署MiniCPM-V教程
本文不生产技术，只做技术的搬运工！！！
参考链接：
ms-swift、minicpm-v
环境准备
ms-swift安装
conda create -n swift python=3.10
conda activate swift
pip install 'ms-swift[all]' -Ullama.cpp安装
conda create -n llama-cpp python=3.10
conda activate llama-cpp
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
pip install -r requirments.txt
make -jollama编译
这里需要使用Minicpm-V官方提供的分支，编译过程中需要下载一些东西，会有网络问题，这个大家自行努力。
git clone -b minicpm-v2.6 https://github.com/OpenBMB/ollama.git
cd ollama/llm
git clone -b minicpmv-main https://github.com/OpenBMB/llama.cpp.git
cd ../
brew install go cmake gcc
go generate ./...
go build .
数据集准备
import os
import json
path = r"your_image_path"
dst = r"xxxxx/data.jsonl"
template = {
"query": "<image>xxxxxxx",
"response": "xxxxxx",
"images": []
}
image_name_list = os.listdir(path)
with open(dst, "w", encoding="utf-8") as f:
for image_name in image_name_list:
template["images"] = [os.path.join(path, image_name)]#这里要写入图像的全部路径，生成后图像不要再移动了
f.write(json.dumps(template, ensure_ascii=False) + "\n")下图为生成好的数据集
微调
#,1,2,3 NPROC_PER_NODE=4
CUDA_VISIBLE_DEVICES=0 swift sft \
--model_type minicpm-v-v2_6-chat \
--model_id_or_path 你模型的具体路径，自行在modelscope上下载，指定到文件夹路径即可 \
--sft_type lora \
--batch_size 1 \
--dataset 使用脚本生成的jsonl文件的具体路径
#--deepspeed default-zero2合并
CUDA_VISIBLE_DEVICES=0 swift export \
--ckpt_dir /data/project_llm/output/minicpm-v-v2_6-chat/v0-20241025-155711/checkpoint-2 \ #这里是你微调后的路径
--merge_lora true转换
这里我只能转换fp16,如果转换官方的Q4_K_M转换会报错，大家可以自行尝试。
注意：这里要分步执行，不要使用shell脚本一起跑
cd /data/project_llm/llama.cpp #切换到你编译的llama.cpp的路径
python /data/project_llm/llama.cpp/examples/llava/minicpmv-surgery.py -m /data/project_llm/output/minicpm-v-v2_6-chat/v0-20241025-155711/checkpoint-2-merged
python /data/project_llm/llama.cpp/examples/llava/minicpmv-convert-image-encoder-to-gguf.py -m /data/project_llm/output/minicpm-v-v2_6-chat/v0-20241025-155711/checkpoint-2-merged --minicpmv-projector /data/project_llm/output/minicpm-v-v2_6-chat/v0-20241025-155711/checkpoint-2-merged/minicpmv.projector --output-dir /data/project_llm/model_finetuned/minicpm-v-2_6/gguf --image-mean 0.5 0.5 0.5 --image-std 0.5 0.5 0.5
python /data/project_llm/llama.cpp/convert_hf_to_gguf.py /data/project_llm/output/minicpm-v-v2_6-chat/v0-20241025-155711/checkpoint-2-merged/model
#/data/project_llm/llama.cpp/llama-quantize /data/project_llm/output/minicpm-v-v2_6-chat/v0-20241025-155711/checkpoint-2-merged/model/Model-7.6B-F16.gguf /data/project_llm/model_finetuned/minicpm-v-2_6/gguf/Model-7.6B-q4_0.gguf q4_0部署
切换到你刚才编译的ollama路径
编写minicpm-v-finetuned.modelfile文件
FROM /data/project_llm/model_finetuned/minicpm-v-2_6/gguf/Model-7.6B-F16.gguf
FROM /data/project_llm/model_finetuned/minicpm-v-2_6/gguf/mmproj-model-f16.gguf
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>{{ end }}
{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>{{ end }}
<|im_start|>assistant<|im_end|>
{{ .Response }}<|im_end|>"""
PARAMETER stop "<|endoftext|>"
PARAMETER stop "<|im_end|>"
PARAMETER num_ctx 2048执行命令
./ollama serve #单独开一个终端
./ollama create minicpm-v-test -f minicpm-v-finetuned.modelfile调用
import base64
from openai import OpenAI
client = OpenAI(base_url='http://localhost:11434/v1', api_key='xxx')
# Function to encode the image
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode('utf-8')
import time
image_path = "/data/pic/20240911091456_20240911095153_P000frame_1849.jpg"
base64_image = encode_image(image_path)
#"url": f"data:image/jpeg;base64,{base64_image}"
start = time.time()
model_type = client.models.list().data[1].id
print(model_type)
response = client.chat.completions.create(
model=model_type,
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "描述一下这张图像"},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}"
}
},
],
}
],
max_tokens=300,
)
print(response.choices[0].message.content)
print(time.time()-start)