当前位置: 首页 > article >正文

GPT3.5\GPT4系列计算完整prompt token数的官方方法

前言:

ChatGPT如何计算token数?https://wtl4it.blog.csdn.net/article/details/135116493?spm=1001.2014.3001.5502icon-default.png?t=N7T8https://wtl4it.blog.csdn.net/article/details/135116493?spm=1001.2014.3001.5502

GPT3.5\GPT4系列计算完整prompt token数的官方方法:

How to count tokens with tiktoken | OpenAI CookbookOpen-source examples and guides for building with the OpenAI API. Browse a collection of snippets, advanced techniques and walkthroughs. Share your own examples and guides.icon-default.png?t=N7T8https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktokenhttps://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynbicon-default.png?t=N7T8https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb

计算的代码如下:
def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613"):
    """Return the number of tokens used by a list of messages."""
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        print("Warning: model not found. Using cl100k_base encoding.")
        encoding = tiktoken.get_encoding("cl100k_base")
    if model in {
        "gpt-3.5-turbo-0613",
        "gpt-3.5-turbo-16k-0613",
        "gpt-4-0314",
        "gpt-4-32k-0314",
        "gpt-4-0613",
        "gpt-4-32k-0613",
        }:
        tokens_per_message = 3
        tokens_per_name = 1
    elif model == "gpt-3.5-turbo-0301":
        tokens_per_message = 4  # every message follows <|start|>{role/name}\n{content}<|end|>\n
        tokens_per_name = -1  # if there's a name, the role is omitted
    elif "gpt-3.5-turbo" in model:
        print("Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.")
        return num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613")
    elif "gpt-4" in model:
        print("Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.")
        return num_tokens_from_messages(messages, model="gpt-4-0613")
    else:
        raise NotImplementedError(
            f"""num_tokens_from_messages() is not implemented for model {model}. See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens."""
        )
    num_tokens = 0
    for message in messages:
        num_tokens += tokens_per_message
        for key, value in message.items():
            num_tokens += len(encoding.encode(value))
            if key == "name":
                num_tokens += tokens_per_name
    num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>
    return num_tokens


http://www.kler.cn/a/227122.html

相关文章:

  • 买电脑注意事项之CPU型号后面的字母都代表什么意思
  • 前端复杂 table 渲染及 excel.js 导出
  • openssl3.2 - helpdoc - P12证书操作
  • ReactNative实现一个圆环进度条
  • 【ELK】logstash快速入门
  • 【乳腺肿瘤诊断分类及预测】基于自适应SPREAD-PNN概率神经网络
  • 单片机学习笔记---定时器/计数器(简述版!)
  • 记录springboot bug
  • SVN单个项目迁移到Gitlab,保留历史提交记录
  • 部署实战--修改jar中的文件并重新打包成jar文件
  • 理想架构的高回退Doherty功率放大器理论与ADS仿真-Multistage
  • springboot153相亲网站
  • JAVA中的多态参数
  • LLM(5) | Encoder 和 Decoder 架构
  • 2.3学习总结
  • Quick BI中lod函数之lod_exclude
  • 【Android Gradle 插件】自定义 Gradle 插件模块 ⑤ ( 完整总结 )
  • idea项目如何上传gitee
  • FCIS 2023:洞悉网络安全新前沿,引领未来安全创新狂潮
  • 2024年美国大学生数学建模比赛MCM问题B:搜索潜水器-思路解析与代码解答