Text2Video Huggingface Pipeline 文生视频接口和文生视频论文API
1. 背景
文生视频是AI领域热点，国内外有非常多的优秀产品如Runway AI、Pika AI、可灵King AI、通义千问、智谱的文生视频模型等等。很多文生视频的大模型都是基于Huggingface的 diffusers的python包来开发。为了方便调用，也尝试了使用 PyPI的text2video的python库的Wrapper类进行调用，下面会给大家介绍一下Huggingface Text to Video Pipeline的调用方式
以及使用通用的text2video的python库调用方式。
2. Huggingface Text to Video Pipeline 代码
地址: (https://huggingface.co/docs/diffusers/api/pipelines/text_to_video)
## code for huggingface diffusion pipeline
import torch
from diffusers import DiffusionPipeline
from diffusers.utils import export_to_videopipe = DiffusionPipeline.from_pretrained("damo-vilab/text-to-video-ms-1.7b", torch_dtype=torch.float16, variant="fp16")
pipe = pipe.to("cuda")prompt = "Spiderman is surfing"
video_frames = pipe(prompt).frames[0]
video_path = export_to_video(video_frames)
video_path
3. 使用Python的包Text2Video来下载最新的文本生成领域论文。
3.1 安装 pip3的 text2video的包
pip install text2video
3.2. 使用现有接口从 arxiv程序化下载最新文生视频的论文
定义输入接口，我们使用的是查询 ArxivPaper的API，需要传入 api_name 字段。同时可以设置 查询接口的 额外属性，包含拓展参数有。
字段|默认值|含义
start| 0|开始entry个数
max_results| 10 | 结束entry个数
sortBy| lastUpdatedDate| 日期字段
sortOrder| descending| 升序或者降序
调用python的 text2video包下载最新发布在 Arxiv论文信息
import text2video as t2v
import jsoninput_dict = {"text": "Text to Video"}
res = t2v.api(input_dict, model=None, api_name="ArxivPaperAPI", start=0, max_results = 3)
paper_list = json.loads(res["text"])
print ("###### Text to Image Recent Paper List:")
for (i, paper_json) in enumerate(paper_list):
print ("|" + paper_json["id"] + "|" + paper_json["title"].replace("\n", "") + "|" + paper_json["updated"] )
输出结果
###### Text to Image Recent Paper List:
|http://arxiv.org/abs/2410.08211v1|LatteCLIP: Unsupervised CLIP Fine-Tuning via LMM-Synthetic Texts|2024-10-10T17:59:59Z
|http://arxiv.org/abs/2410.08210v1|PointOBB-v2: Towards Simpler, Faster, and Stronger Single Point Supervised Oriented Object Detection|2024-10-10T17:59:56Z
|http://arxiv.org/abs/2410.08209v1|Emerging Pixel Grounding in Large Multimodal Models Without Grounding Supervision|2024-10-10T17:59:55Z
3.3 自定义接口实现text2Video的API Wrapper
继承类 BaseAPI
入参
字段|类型|含义
input_dict| 字典| 处理API输入 text,image,audio,video字段
model| Huggingface的模型 Pytorch
kwargs| 额外参数
出参
output_dict| 字典| API输出的结果的dict,包含4个key text,image,audio,video字段
核心逻辑
model继承自 Huggingface的 text_to_video的 pipeline (https://huggingface.co/docs/diffusers/api/pipelines/text_to_video)
4. 相关代码库 Github/PyPI地址
https://github.com/rockingdingo/text2video
https://github.com/rockingdingo/text2audio
https://github.com/rockingdingo/image2video
https://github.com/rockingdingo/SuperAlignment
https://github.com/rockingdingo/SuperIntelligence
http://www.deepnlp.org/blog/introduction-to-text-to-video-generation-huggingface-pipeline-and-pypi-package-text2video
http://www.deepnlp.org/blog/introduction-to-multimodal-generative-modelshttps://huggingface.co/docs/diffusers/api/pipelines/text_to_video
5. AI应用用户评价User Reviews
OpenAI o1
http://www.deepnlp.org/store/pub/pub-openai-o1
ChatGPT User Reviews
http://www.deepnlp.org/store/pub/pub-chatgpt-openai
Gemini User Reviews
http://www.deepnlp.org/store/pub/pub-gemini-google
Perplexity User Reviews
http://www.deepnlp.org/store/pub/pub-perplexity
Claude User Reviews
http://www.deepnlp.org/store/pub/pub-claude-anthropic
Grok User Reviews
http://www.deepnlp.org/store/pub/pub-grok-xai
Midjourney User Reviews
http://www.deepnlp.org/store/pub/pub-midjourney
Stable Diffusion User Reviews
http://www.deepnlp.org/store/pub/pub-stable-diffusion
Runway User Reviews
Canva User Reviews
http://www.deepnlp.org/store/pub/pub-canva
GPT-5 Forecast
http://www.deepnlp.org/store/pub/pub-gpt-5
SearchGPT Reviews
http://www.deepnlp.org/store/pub/pub-searchgpt
Kling AI Reviews
http://www.deepnlp.org/store/pub/pub-kling-kwai
Dreamina AI Reviews
http://www.deepnlp.org/store/pub/pub-dreamina-douyin
Luma AI
http://www.deepnlp.org/store/pub/pub-luma-ai
Pika AI Reviews
http://www.deepnlp.org/store/pub/pub-pika
Runway AI Reviews
http://www.deepnlp.org/store/pub/pub-runway
Flux AI Reviews
http://www.deepnlp.org/store/pub/pub-flux-1-black-forest-lab
Qwen AI Reviews
http://www.deepnlp.org/store/pub/pub-qwen-alibaba
Zhipu AI Reviews
http://www.deepnlp.org/store/pub/pub-zhipu-ai
Doubao Reviews
http://www.deepnlp.org/store/pub/pub-doubao-douyin
Kimi Chat Reviews
http://www.deepnlp.org/store/pub/pub-kimi-ai
Coursera Reviews
http://www.deepnlp.org/store/pub/pub-coursera
Udacity Reviews
http://www.deepnlp.org/store/pub/pub-udacity
Grammarly Reviews
http://www.deepnlp.org/store/pub/pub-grammarly
ChatGPT Strawberry
http://www.deepnlp.org/store/pub/pub-chatgpt-strawberry
Google AR VR Headsets
http://www.deepnlp.org/store/pub/pub-google-ar-vr-headset
DeepNLP AI Tools
http://www.deepnlp.org/store/pub/pub-deepnlp-ai
## Robotics
Tesla Cybercab Robotaxi
http://www.deepnlp.org/store/pub/pub-tesla-cybercab
Tesla Optimus
http://www.deepnlp.org/store/pub/pub-tesla-optimus
Figure AI
http://www.deepnlp.org/store/pub/pub-figure-ai
Unitree Robotics Reviews
http://www.deepnlp.org/store/pub/pub-unitree-robotics
Waymo User Reviews
http://www.deepnlp.org/store/pub/pub-waymo-google
ANYbotics Reviews
http://www.deepnlp.org/store/pub/pub-anybotics
Boston Dynamics
http://www.deepnlp.org/store/pub/pub-boston-dynamic
## AI Widgets
Apple Glasses
http://www.deepnlp.org/store/pub/pub-apple-glasses
Meta Glasses
http://www.deepnlp.org/store/pub/pub-meta-glasses
Apple AR VR Headset
http://www.deepnlp.org/store/pub/pub-apple-ar-vr-headset
Google Glass
http://www.deepnlp.org/store/pub/pub-google-glass
Meta VR Headset
http://www.deepnlp.org/store/pub/pub-meta-vr-headset
## Social
Character AI
http://www.deepnlp.org/store/pub/pub-character-ai
## Self-Driving
BYD Seal
http://www.deepnlp.org/store/pub/pub-byd-seal
Tesla Model 3
http://www.deepnlp.org/store/pub/pub-tesla-model-3
BMW i4
http://www.deepnlp.org/store/pub/pub-bmw-i4
Baidu Apollo Reviews
http://www.deepnlp.org/store/pub/pub-baidu-apollo
Hyundai IONIQ 6
http://www.deepnlp.org/store/pub/pub-hyundai-ioniq-6