GPT实战系列-简单聊聊LangChain搭建本地知识库准备

本文介绍: 利用一些文档中的信息来构建答案，回答特定文档中的问题。首先需要把文本转换为文本向量，即自然语言处理常常要用的Embedding技术，Text2Vector。

LangChain 是一个开发由语言模型驱动的应用程序的框架，除了和应用程序通过 API 调用，还会：

LLM大模型相关文章：

GPT实战系列-ChatGLM3本地部署CUDA11+1080Ti+显卡24G实战方案

from langchain.embeddings import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()

from langchain.embeddings import HuggingFaceEmbeddings, SentenceTransformerEmbeddings

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
# Equivalent to SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")

from langchain.embeddings import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings()

from langchain.embeddings import ModelScopeEmbeddings

model_id = "damo/nlp_corom_sentence-embedding_english-base"

embeddings = ModelScopeEmbeddings(model_id=model_id)

from langchain.embeddings import TensorflowHubEmbeddings

embeddings = TensorflowHubEmbeddings()

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import FAISS
from langchain.document_loaders import TextLoader

from langchain.document_loaders import TextLoader

loader = TextLoader("../../../state_of_the_union.txt")
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
docs = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings()

db = FAISS.from_documents(docs, embeddings)