当前位置：首页 > article >正文

基于ragflow中deepdoc对pdf文档的rag系统

article 2025/3/15 0:46:50

基于ragflow中deepdoc对pdf文档的rag系统

一、安装

conda环境安装到指定的路径

conda create 包名/环境的名字
在这里插入图片描述
rag就是包的名字，ragflow就是环境名；

怎样激活环境？–我是在百度飞桨上面跑的
conda activate /home/aistudio/rag/ragflow

安装python库

datrie==0.8.2
akshare>=1.15.78,<2.0.0
azure-storage-blob==12.22.0
azure-identity==1.17.1
azure-storage-file-datalake==12.16.0
anthropic==0.34.1
arxiv==2.1.3
aspose-slides>=24.9.0,<25.0.0; platform_machine == 'x86_64' or (sys_platform == 'darwin' and platform_machine == 'arm64')
beartype>=0.18.5,<0.19.0
bio==1.7.1
blinker==1.7.0
boto3==1.34.140
botocore==1.34.140
cachetools==5.3.3
chardet==5.2.0
cn2an==0.5.22
cohere==5.6.2
Crawl4AI==0.3.8
dashscope==1.20.11
deepl==1.18.0
demjson3==3.0.6
discord-py==2.3.2
duckduckgo-search>=7.2.0,<8.0.0
editdistance==0.8.1
elastic-transport==8.12.0
elasticsearch==8.12.1
elasticsearch-dsl==8.12.0
fasttext==0.9.3
filelock==3.15.4
flask==3.0.3
flask-cors==5.0.0
flask-login==0.6.3
flask-session==0.8.0
google-search-results==2.4.2
groq==0.9.0
hanziconv==0.3.2
html-text==0.6.2
httpx==0.27.0
huggingface-hub>=0.25.0,<0.26.0
infinity-sdk==0.6.0-dev3
infinity-emb>=0.0.66,<0.0.67
itsdangerous==2.1.2
json-repair==0.35.0
markdown==3.6
markdown-to-json==2.1.1
minio==7.2.4
mistralai==0.4.2
nltk==3.9.1
numpy>=1.26.0,<2.0.0
ollama==0.2.1
onnxruntime==1.19.2; sys_platform == 'darwin' or platform_machine != 'x86_64'
onnxruntime-gpu==1.19.2; sys_platform != 'darwin' and platform_machine == 'x86_64'
openai==1.45.0
opencv-python==4.10.0.84
opencv-python-headless==4.10.0.84
openpyxl>=3.1.0,<4.0.0
ormsgpack==1.5.0
pandas>=2.2.0,<3.0.0
pdfplumber==0.10.4
peewee==3.17.1
pillow==10.4.0
protobuf==5.27.2
psycopg2-binary==2.9.9
pyclipper==1.3.0.post5
pycryptodomex==3.20.0
pypdf>=5.0.0,<6.0.0
pytest>=8.3.0,<9.0.0
python-dotenv==1.0.1
python-dateutil==2.8.2
python-pptx>=1.0.2,<2.0.0
pywencai==0.12.2
qianfan==0.4.6
ranx==0.3.20
readability-lxml==0.8.1
valkey==6.0.2
requests==2.32.2
replicate==0.31.0
roman-numbers==1.0.2
ruamel-base==1.0.0
scholarly==1.7.11
scikit-learn==1.5.0
selenium==4.22.0
selenium-wire==5.1.0
setuptools>=75.2.0,<76.0.0
shapely==2.0.5
six==1.16.0
strenum==0.4.15
tabulate==0.9.0
tencentcloud-sdk-python==3.0.1215
tika==2.6.0
tiktoken==0.7.0
umap_learn==0.5.6
vertexai==1.64.0
volcengine==1.0.146
voyageai==0.2.3
webdriver-manager==4.0.1
werkzeug==3.0.6
wikipedia==1.4.0
word2number==1.1
xgboost==1.5.0
xpinyin==0.7.6
yfinance==0.1.96
zhipuai==2.0.1
ruamel-yaml>=0.18.6,<0.19.0
google-generativeai>=0.8.1,<0.9.0
python-docx>=1.1.2,<2.0.0
pypdf2>=3.0.1,<4.0.0
graspologic>=3.4.1,<4.0.0
pymysql>=1.1.1,<2.0.0
mini-racer>=0.12.4,<0.13.0
pyodbc>=5.2.0,<6.0.0
flasgger>=0.9.7.1,<0.10.0
xxhash>=3.5.0,<4.0.0
bcembedding==0.1.5
fastembed>=0.3.6,<0.4.0; sys_platform == 'darwin' or platform_machine != 'x86_64'
fastembed-gpu>=0.3.6,<0.4.0; sys_platform != 'darwin' and platform_machine == 'x86_64'
flagembedding==1.2.10
torch>=2.5.0,<3.0.0
transformers>=4.35.0,<5.0.0
langchain
langchain-community
faiss-gpu

怎样从pyproject.toml中使用conda进行安装；
- 直接调用大模型上传pyproject.toml加上提示词就能够生成requirements.txt

注意：版本有问题，使用最新版本的进行安装的pyproject.toml进行改装；

二、解析pdf文件

根据ragflow的demo中可以查看解析的格式

https://demo.ragflow.io
在这里插入图片描述

到达路径下面-`cd ragflow/rag/app/`接可以看到可以解析的各种切片的方法：

在这里插入图片描述

怎样进行切分呢？（基本上只有这些）

+ 法律文档：使用laws.py来运行
+ ppt：presentation.py
+ 论文：paper.py
+ 简历：resume.py
+ 问答：qa.py

都可以在https://demo.ragflow.io中查看简介

以paper.py为例

在import前面添加：

import sys
sys.path.append("../..")

在这里插入图片描述
2. 在主函数后面直接替换

if __name__ == "__main__":
    import sys
    def dummy(prog=None, msg=""):
        print(prog,msg)
    #chunk(sys.argv[1], callback=dummy)
    import os
    path = "/home/aistudio/majority"
    files=os.listdir(path)
    file_paths=[path+"/"+file for file in files]
    print("file_paths=",file_paths)
    for i in file_paths:
        print(i)
    all_chunks = []
    # 遍历处理每个文件
    for i,path in enumerate(file_paths):
        print(f"第{i}path=",path)
        chunks = chunk(path, callback=dummy)
        with open("/home/aistudio/paper.txt","a") as f:
            for chunk1 in chunks:
                f.write(chunk1["content_with_weight"]+"\n\n")
        all_chunks.extend(chunks)  # 合并所有chunks
        print(f"path={path}成功")
    print("Total chunks=", len(all_chunks))
    # 保存合并后的txt文件
    
    from langchain.embeddings import HuggingFaceEmbeddings
    from langchain.vectorstores import FAISS
    embeddings = HuggingFaceEmbeddings(
        model_name="/home/aistudio/bge_large",
        model_kwargs = {'device': 'cuda'}
    )
    chunks_content = [chunk["content_with_weight"] for chunk in all_chunks]
    FAISS.from_texts(chunks_content, embeddings).save_local(r"/home/aistudio/faiss_ppt")
    print("多文件保存完毕")

运行
在ragflow/rag/app的路径下运行：

python paper.py

注意 path = "/home/aistudio/majority"是pdf的文件夹的路径；

三、rag检索向量数据库

def retriever(query):
    #读取faiss_ppt的数据库
    from langchain.embeddings import HuggingFaceEmbeddings
    from langchain.vectorstores import FAISS

    embeddings = HuggingFaceEmbeddings(model_name=r"D:\program\phone_check\bge_large")
    db = FAISS.load_local("faiss_ppt", embeddings,allow_dangerous_deserialization = True)
    # 从数据库中获取与问题最相关的10个的文档
    docs = db.similarity_search(query, k=10)
    #读取docs的内容
    docs_content=[]
    for doc in docs:
        docs_content.append(doc.page_content)
    return docs_content

query:是字符串
model_name:是本地的模型路径（保存faiss的embedding与加载faiss的我embedding相同）