I developed a tool called Know Me that answers questions about you based on information from your website and CV. Below, you’ll find a video that provides an overview of this tool. In this post, I’ll walk you through the development process and discuss some of the technical aspects involved.
[https://drive.google.com/file/d/102byW3PA_4WtQHwrfpZHX25Nkvq-LdhE/view?usp=sharing](https://drive.google.com/file/d/102byW3PA_4WtQHwrfpZHX25Nkvq-LdhE/view?usp=sharing)
<aside> 💡 Feel free to skip this section if you are already familiar with retrieval-augmented generation (RAG).
</aside>
Current large language models (LLMs) are trained on general domain data to acquire broad expertise. However, this approach has several drawbacks. First, the training data may be outdated, limiting the LLM's ability to answer questions or perform tasks requiring the latest information. Additionally, LLMs may struggle to handle tasks involving proprietary or private data. Retrieval-augmented generation (RAG) addresses these limitations by integrating retrieval mechanisms to access up-to-date and specific information, enhancing the model's capabilities and relevance.
Retrieval Augmented Generation (RAG) is a technique that combines the strengths of retrieval-based and generation-based models in natural language processing. It involves using a retrieval system to find relevant documents or pieces of information and then leveraging a generation model to produce a coherent and contextually appropriate response based on the retrieved information. Here’s a more detailed breakdown:
• In the first phase, a retrieval model (often based on search algorithms or embeddings) is used to fetch relevant documents or data from a large corpus. The retrieval model is typically trained to identify the most relevant pieces of text that can help answer a given query.
• After retrieving the relevant documents, a generative model, such as a transformer-based model (e.g., GPT-3), is employed to generate a response. The generative model uses the context provided by the retrieved documents to produce a more accurate and informative output.
Here are the different components that were used to build the KnowMe Tool. I will explain what the different components do and the decisions that I have taken along the way
The decisions that I took are indicated in a callout like this
<aside> ⚖️ This indicates a decision that I took
</aside>
Building a history aware Chain (using Langchain).