transformers gpt2 语言模型
一、安装依赖
pip install transformers
pip install torch
pip install SentencePiece二、下载模型
https://huggingface.co/openai-community/gpt2/tree/main
拷贝到目录：
运行代码：
from transformers import GPT2Tokenizer, GPT2LMHeadModel, pipeline
# 加载预训练的模型和分词器
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
# 使用pipeline简化流程
generator = pipeline("text-generation", model=model, tokenizer=tokenizer)
# 文本生成
prompt = "the world is big? "
generated_text = generator(prompt, max_length=50, num_beams=5)
print(generated_text[0]['generated_text'])
得到结果：
the world is big? I don't think so. In fact, I think we're going to see a lot more of that in the next couple of years. I don't think we're going to see a lot
