mirror of
https://github.com/ArthurDanjou/ArtStudies.git
synced 2026-03-16 03:11:46 +01:00
2267 lines
1.5 MiB
2267 lines
1.5 MiB
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 🏅 RAG Olympics - Building an AI-Powered Q&A System!\n",
|
||
"\n",
|
||
"## 🎯 Welcome to the RAG Challenge!\n",
|
||
"\n",
|
||
"In this hands-on project, you'll build a **Retrieval-Augmented Generation (RAG)** system that can answer questions about the Olympics!\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### 🤔 What is RAG?\n",
|
||
"\n",
|
||
"**RAG = Retrieval + Generation**\n",
|
||
"\n",
|
||
"Think of it like having a super-smart assistant with access to a library:\n",
|
||
"\n",
|
||
"```\n",
|
||
"User Question → 📚 Find Relevant Documents → 🤖 LLM Generates Answer\n",
|
||
"```\n",
|
||
"\n",
|
||
"**The Magic:** Instead of relying only on the LLM's training data, we give it **real-time access to specific information**!\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### 🎓 What You'll Learn:\n",
|
||
"\n",
|
||
"| Step | What | Why |\n",
|
||
"|------|------|-----|\n",
|
||
"| 1️⃣ | **Load Data** | Get Olympics FAQ documents |\n",
|
||
"| 2️⃣ | **Create Embeddings** | Convert text to vectors |\n",
|
||
"| 3️⃣ | **Build Vector DB** | Store for fast similarity search |\n",
|
||
"| 4️⃣ | **Retrieval** | Find relevant documents |\n",
|
||
"| 5️⃣ | **Prompt Engineering** | Craft effective prompts |\n",
|
||
"| 6️⃣ | **Generate Answers** | Use LLM with context |\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### 🌟 Why RAG is useful ?\n",
|
||
"\n",
|
||
"✅ **Up-to-date info:** Add new knowledge without retraining \n",
|
||
"✅ **Grounded answers:** Responses based on your documents \n",
|
||
"✅ **Reduced hallucinations:** LLM has concrete context \n",
|
||
"✅ **Transparent sources:** Can cite which documents were used \n",
|
||
"✅ **Cost-effective:** No need to fine-tune large models \n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### 🏗️ System Architecture:\n",
|
||
"\n",
|
||
"```\n",
|
||
"┌─────────────────────────────────────────────────────────┐\n",
|
||
"│ RAG PIPELINE │\n",
|
||
"├─────────────────────────────────────────────────────────┤\n",
|
||
"│ │\n",
|
||
"│ 1. INDEXING PHASE (One-time setup) │\n",
|
||
"│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │\n",
|
||
"│ │ Documents│ -> │Embeddings│ -> │ Vector DB│ │\n",
|
||
"│ └──────────┘ └──────────┘ └──────────┘ │\n",
|
||
"│ │\n",
|
||
"│ 2. QUERY PHASE (Real-time) │\n",
|
||
"│ ┌─────────┐ ┌──────────┐ ┌──────────┐ ┌─────┐ │\n",
|
||
"│ │ Question│ ->│ Search DB│ ->│ Context │ ->│ LLM │ │\n",
|
||
"│ └─────────┘ └──────────┘ └──────────┘ └─────┘ │\n",
|
||
"│ ↓ │\n",
|
||
"│ ┌────────┐│\n",
|
||
"│ │ Answer ││\n",
|
||
"│ └────────┘│\n",
|
||
"└─────────────────────────────────────────────────────────┘\n",
|
||
"```\n",
|
||
"\n",
|
||
"Let's build it! 🚀"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 📄 Step 1: Loading the Olympics FAQ Data\n",
|
||
"\n",
|
||
"We'll work with a JSON file containing frequently asked questions about the Olympics.\n",
|
||
"\n",
|
||
"### 📊 Data Structure:\n",
|
||
"\n",
|
||
"```json\n",
|
||
"[\n",
|
||
" {\n",
|
||
" \"id\": \"OnlOaClN-fr\",\n",
|
||
" \"lang\": \"fr\",\n",
|
||
" \"label\": \"Comment acheter des billets pour...\",\n",
|
||
" \"body\": \"Des billets pour les Jeux Olympiques...\",\n",
|
||
" \"topics\": \"Billetterie et Hospitalité...\", \n",
|
||
" \"url\": \"https://help.paris2024.org/contents...\"\n",
|
||
" },\n",
|
||
" ...\n",
|
||
"]\n",
|
||
"```\n",
|
||
"\n",
|
||
"Let's load and explore the data!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"✅ Loaded 952 FAQ entries!\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Load the FAQ data\n",
|
||
"import json\n",
|
||
"from pathlib import Path\n",
|
||
"\n",
|
||
"# Get the notebook's directory\n",
|
||
"notebook_dir = Path.cwd()\n",
|
||
"data_path = notebook_dir / \"data\" / \"paris-2024-faq.json\"\n",
|
||
"with Path(data_path).open() as f:\n",
|
||
" faq_data = json.load(f)\n",
|
||
"\n",
|
||
"print(f\"✅ Loaded {len(faq_data)} FAQ entries!\\n\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"{'id': '-Bgz0H-g-fr',\n",
|
||
" 'lang': 'fr',\n",
|
||
" 'label': 'Existe-t-il des places spécifiques pour les personnes avec un handicap mental/neurologique/psychique ?',\n",
|
||
" 'body': \"Les personnes avec un handicap mental, neurologique ou psychologique peuvent bénéficier des places réservées pour les personnes en situation de handicap qui sont faciles d'accès sur présentation d'une carte d'invalidité ou d'un équivalent.\\n\\nTous les volontaires seront sensibilisés au handicap mental, neurologique et psychique afin de garantir la meilleure expérience et le meilleur accueil possible.\",\n",
|
||
" 'topics': 'Spectateurs ;Accessibilité;Accès / services sur site',\n",
|
||
" 'url': 'https://help.paris2024.org/contents/Existe-t-il-des-places-specifiques-pour-les-personnes-avec-un-handicap-mental-neurologique-psychique--Bgz0H-g'}"
|
||
]
|
||
},
|
||
"execution_count": 10,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Display first entry\n",
|
||
"faq_data[0]\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Filter to keep only English documents\n",
|
||
"faq_data_en = [item for item in faq_data if item[\"lang\"] == \"en\"]\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"📋 Sample FAQ Entries:\n",
|
||
"\n",
|
||
"1. Q: How often will the toilets at the competition venues be cleaned?\n",
|
||
" A: The toilets will be cleaned regularly during each session....\n",
|
||
"\n",
|
||
"2. Q: How do you apply to carry the Flame?\n",
|
||
" A: The public campaigns for torchbearers have all concluded.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"For more information, please visit:\n",
|
||
"\n",
|
||
"ht...\n",
|
||
"\n",
|
||
"3. Q: Combien y a-t-il de volontaires ?\n",
|
||
" A: The deadline for applying to the Paris 2024 volunteering programme has already passed.\n",
|
||
"\n",
|
||
"However, the...\n",
|
||
"\n",
|
||
"💡 Total entries: 478\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Display sample entries\n",
|
||
"print(\"📋 Sample FAQ Entries:\\n\")\n",
|
||
"for i, entry in enumerate(faq_data_en[:3], 1):\n",
|
||
" print(f\"{i}. Q: {entry['label']}\")\n",
|
||
" print(f\" A: {entry['body'][:100]}...\\n\")\n",
|
||
"\n",
|
||
"print(f\"💡 Total entries: {len(faq_data_en)}\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 🔄 Step 2: Preprocessing - Creating LangChain Documents\n",
|
||
"\n",
|
||
"### Why Create Documents?\n",
|
||
"\n",
|
||
"LangChain uses a special `Document` object that:\n",
|
||
"- 📝 Stores the text content\n",
|
||
"- 🏷️ Includes metadata (source, category, etc.)\n",
|
||
"- 🔍 Works seamlessly with vector stores\n",
|
||
"\n",
|
||
"First get familiar with this object"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"page_content='Hello, world!' metadata={'source': 'https://example.com'}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from langchain_core.documents import Document\n",
|
||
"\n",
|
||
"# The LangChain Document object contains 2 attributes:\n",
|
||
"# - a page_content attribute: stores the main text content\n",
|
||
"# - a metadata attribute: stores additional information about the document\n",
|
||
"\n",
|
||
"\n",
|
||
"# Creating a sample Document object\n",
|
||
"document = Document(\n",
|
||
" page_content=\"Hello, world!\", metadata={\"source\": \"https://example.com\"},\n",
|
||
")\n",
|
||
"\n",
|
||
"# Display the document\n",
|
||
"print(document)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"'Hello, world!'"
|
||
]
|
||
},
|
||
"execution_count": 14,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"document.page_content\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"{'source': 'https://example.com'}"
|
||
]
|
||
},
|
||
"execution_count": 15,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"document.metadata\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Our Strategy:\n",
|
||
"\n",
|
||
"We'll combine each Q&A pair into a single document:\n",
|
||
"```\n",
|
||
"\"Question: [question]\\nAnswer: [answer]\"\n",
|
||
"```\n",
|
||
"\n",
|
||
"This helps the retrieval system find relevant Q&A pairs based on user questions!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"🔄 Creating LangChain Document objects...\n",
|
||
"\n",
|
||
"✅ Created 478 Document objects!\n",
|
||
"\n",
|
||
"📄 Example Document Structure:\n",
|
||
"\n",
|
||
"Content (first 200 chars):\n",
|
||
"Question: How often will the toilets at the competition venues be cleaned?\n",
|
||
"Answer: The toilets will be cleaned regularly during each session....\n",
|
||
"\n",
|
||
"Metadata:\n",
|
||
"{'source': 'Olympics FAQ', 'faq_id': 'OnlBYpRa-en', 'question': 'How often will the toilets at the competition venues be cleaned?', 'answer': 'The toilets will be cleaned regularly during each session.', 'topics': 'Other;Environmental commitments;Waste management', 'url': 'https://help.paris2024.org/en-gb/contents/How-often-will-the-toilets-at-the-competition-venues-be-cleaned-OnlBYpRa'}\n",
|
||
"\n",
|
||
"💡 These documents are now ready for embedding!\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(\"🔄 Creating LangChain Document objects...\\n\")\n",
|
||
"\n",
|
||
"# Convert FAQ entries to LangChain Documents\n",
|
||
"documents = []\n",
|
||
"\n",
|
||
"for entry in faq_data_en:\n",
|
||
" # Combine question and answer into one document\n",
|
||
" content = f\"Question: {entry['label']}\\nAnswer: {entry['body']}\"\n",
|
||
"\n",
|
||
" # Create Document with metadata\n",
|
||
" doc = Document(\n",
|
||
" page_content=content,\n",
|
||
" metadata={\n",
|
||
" \"source\": \"Olympics FAQ\",\n",
|
||
" \"faq_id\": entry[\"id\"],\n",
|
||
" \"question\": entry[\"label\"],\n",
|
||
" \"answer\": entry[\"body\"],\n",
|
||
" \"topics\": entry[\"topics\"],\n",
|
||
" \"url\": entry[\"url\"],\n",
|
||
" },\n",
|
||
" )\n",
|
||
"\n",
|
||
" documents.append(doc)\n",
|
||
"\n",
|
||
"print(f\"✅ Created {len(documents)} Document objects!\\n\")\n",
|
||
"\n",
|
||
"# Show an example document\n",
|
||
"print(\"📄 Example Document Structure:\\n\")\n",
|
||
"print(f\"Content (first 200 chars):\\n{documents[0].page_content[:200]}...\\n\")\n",
|
||
"print(f\"Metadata:\\n{documents[0].metadata}\")\n",
|
||
"\n",
|
||
"print(\"\\n💡 These documents are now ready for embedding!\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 🧠 Step 4: Creating the Vector Database\n",
|
||
"\n",
|
||
"This is where the magic happens! We'll:\n",
|
||
"1. Convert documents to embeddings (vectors)\n",
|
||
"2. Store them in a Qdrant vector database\n",
|
||
"3. Enable lightning-fast similarity search\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### 🎯 What is Qdrant?\n",
|
||
"\n",
|
||
"**Qdrant** (read: quadrant) is a vector similarity search engine and database:\n",
|
||
"- ⚡ **Fast:** High-performance search with advanced filtering capabilities\n",
|
||
"- 💪 **Powerful:** Production-ready with built-in scalability and clustering\n",
|
||
"- 🎨 **Flexible:** Supports payloads, filtering, and multiple distance metrics\n",
|
||
"- 🆓 **Open Source:** Free to use with both in-memory and persistent storage options\n",
|
||
"- 🐳 **Cloud-Native:** Easy deployment with Docker, Kubernetes, or managed cloud service\n",
|
||
"- 🚀 **Versatile Deployment:**\n",
|
||
" - **In-Memory:** Perfect for development and testing\n",
|
||
" - **Docker:** Quick local setup with containerization\n",
|
||
" - **Local:** Run natively on your machine for production\n",
|
||
" - **Cloud:** Managed service (Qdrant Cloud) for scalability\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### 🔤 Embedding Model:\n",
|
||
"\n",
|
||
"We'll use **mistral-embed** (Mistral AI):\n",
|
||
"- 📊 1024-dimensional vectors\n",
|
||
"- Max input: 8192 tokens\n",
|
||
"- ⚡ Fast encoding speed\n",
|
||
"- 🎯 Great for semantic search\n",
|
||
"- 🌐 Works well in multiple languages\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"import os\n",
|
||
"\n",
|
||
"os.environ[\"MISTRAL_API_KEY\"] = \"YvbkD5LcIf4GJOiEA1xkjr7ZEjUtZjxN\"\n",
|
||
"\n",
|
||
"\n",
|
||
"# Initialize the embedding model\n",
|
||
"from langchain_mistralai import MistralAIEmbeddings\n",
|
||
"\n",
|
||
"embeddings = MistralAIEmbeddings(\n",
|
||
" model=\"mistral-embed\",\n",
|
||
")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 18,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain_qdrant import QdrantVectorStore\n",
|
||
"from qdrant_client import QdrantClient\n",
|
||
"from qdrant_client.http.models import Distance, VectorParams\n",
|
||
"\n",
|
||
"# client = QdrantClient(path=local_output)\n",
|
||
"# In memory client:\n",
|
||
"COLLECTION_NAME = \"olympics-2024\"\n",
|
||
"# VECTOR_NAME = \"sparse-embedding\" # this is to asspciate a name with the embedding\n",
|
||
"EMBEDDING_SIZE = len(embeddings.embed_query(\"hello\"))\n",
|
||
"\n",
|
||
"# instanciate the in memory client\n",
|
||
"client = QdrantClient(\":memory:\")\n",
|
||
"client.create_collection(\n",
|
||
" collection_name=COLLECTION_NAME,\n",
|
||
" vectors_config=VectorParams(size=EMBEDDING_SIZE, distance=Distance.COSINE),\n",
|
||
")\n",
|
||
"\n",
|
||
"vector_store = QdrantVectorStore(\n",
|
||
" client=client,\n",
|
||
" embedding=embeddings,\n",
|
||
" collection_name=COLLECTION_NAME,\n",
|
||
")\n",
|
||
"\n",
|
||
"# See documentation https://reference.langchain.com/python/integrations/langchain_qdrant/\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 19,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"============================================================\n",
|
||
"💡 What just happened:\n",
|
||
" 1. Each document was converted to a 1024D vector\n",
|
||
" 2. Vectors were indexed in Qdrant for fast search\n",
|
||
" 3. Now we can find similar documents!\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from uuid import uuid4\n",
|
||
"\n",
|
||
"ids = [str(uuid4()) for _ in range(len(documents))]\n",
|
||
"vector_store.add_documents(documents=documents, ids=ids)\n",
|
||
"\n",
|
||
"\n",
|
||
"print(\"\\n\" + \"=\" * 60)\n",
|
||
"print(\"💡 What just happened:\")\n",
|
||
"print(\" 1. Each document was converted to a 1024D vector\")\n",
|
||
"print(\" 2. Vectors were indexed in Qdrant for fast search\")\n",
|
||
"print(\" 3. Now we can find similar documents!\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 🔍 Step 5: Testing the Retrieval System\n",
|
||
"\n",
|
||
"Let's test our vector database by searching for relevant documents!\n",
|
||
"\n",
|
||
"### How Retrieval Works:\n",
|
||
"\n",
|
||
"```\n",
|
||
"User Query → Embed Query → Find Similar Vectors → Return Documents\n",
|
||
"```\n",
|
||
"\n",
|
||
"The system uses **cosine similarity** to find the most relevant documents!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 20,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"🔍 Testing the Retrieval System\n",
|
||
"\n",
|
||
"============================================================\n",
|
||
"❓ User Query: 'How do I get disabled parking spaces?'\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(\"🔍 Testing the Retrieval System\\n\")\n",
|
||
"print(\"=\" * 60)\n",
|
||
"\n",
|
||
"# Test query\n",
|
||
"test_query = \"How do I get disabled parking spaces?\"\n",
|
||
"\n",
|
||
"print(f\"❓ User Query: '{test_query}'\\n\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 21,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"✅ Found 3 relevant documents!\n",
|
||
"\n",
|
||
"📄 Retrieved Documents:\n",
|
||
"\n",
|
||
"============================================================\n",
|
||
"Document 1:\n",
|
||
"============================================================\n",
|
||
"Content:\n",
|
||
"Question: Are parking spaces reserved for individuals with disabilities available on-site?\n",
|
||
"Answer: Each site will have a limited number of parking spaces reserved for wheelchair users.\n",
|
||
"\n",
|
||
"Holders of a PFR ticket will be able to access them without reservation and subject to availability, upon presentation of a European parking card, a mobility inclusion card, and a PFR ticket.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"For countries not issuing either of these two cards, any other official document justifying a disability will be accepted.\n",
|
||
"\n",
|
||
"============================================================\n",
|
||
"Document 2:\n",
|
||
"============================================================\n",
|
||
"Content:\n",
|
||
"Question: What documentary evidence do visitors with disabilities from abroad need?\n",
|
||
"Answer: Visitors will need to provide documentation to book PFR (wheelchair spaces) or PSH (easy access seating in the stands).\n",
|
||
"\n",
|
||
"This documentation can be a European parking card or a mobility inclusion card.\n",
|
||
"\n",
|
||
"For countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"For countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.\n",
|
||
"\n",
|
||
"============================================================\n",
|
||
"Document 3:\n",
|
||
"============================================================\n",
|
||
"Content:\n",
|
||
"Question: Are there any specific spaces for individuals with mental/neurological/psychological disabilities?\n",
|
||
"Answer: Individuals with mental, neurological, or psychological disabilities can access reserved spots for people with disabilities that are easily accessible upon presentation of a disability card or equivalent.\n",
|
||
"\n",
|
||
"All volunteers will receive training on mental, neurological, and psychological disabilities to ensure the best possible experience and welcome.\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Retrieve top 3 most relevant documents\n",
|
||
"retrieved_docs = vector_store.similarity_search(\n",
|
||
" query=test_query, k=3, # Return top 3 matches\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"✅ Found {len(retrieved_docs)} relevant documents!\\n\")\n",
|
||
"print(\"📄 Retrieved Documents:\\n\")\n",
|
||
"\n",
|
||
"for i, doc in enumerate(retrieved_docs, 1):\n",
|
||
" print(f\"{'='*60}\")\n",
|
||
" print(f\"Document {i}:\")\n",
|
||
" print(f\"{'='*60}\")\n",
|
||
" print(f\"Content:\\n{doc.page_content}\\n\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 27,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"[(Document(metadata={'source': 'Olympics FAQ', 'faq_id': 'OnlmLzuE-en', 'question': 'Are parking spaces reserved for individuals with disabilities available on-site?', 'answer': 'Each site will have a limited number of parking spaces reserved for wheelchair users.\\n\\nHolders of a PFR ticket will be able to access them without reservation and subject to availability, upon presentation of a European parking card, a mobility inclusion card, and a PFR ticket.\\n\\n\\n\\nFor countries not issuing either of these two cards, any other official document justifying a disability will be accepted.', 'topics': 'Spectators;Accessibility;On-site access / services', 'url': 'https://help.paris2024.org/en-gb/contents/Are-parking-spaces-reserved-for-individuals-with-disabilities-available-on-site-OnlmLzuE', '_id': 'be6db04a-2902-4ccd-bc29-d2f0d010fff4', '_collection_name': 'olympics-2024'}, page_content='Question: Are parking spaces reserved for individuals with disabilities available on-site?\\nAnswer: Each site will have a limited number of parking spaces reserved for wheelchair users.\\n\\nHolders of a PFR ticket will be able to access them without reservation and subject to availability, upon presentation of a European parking card, a mobility inclusion card, and a PFR ticket.\\n\\n\\n\\nFor countries not issuing either of these two cards, any other official document justifying a disability will be accepted.'),\n",
|
||
" 0.8989433697194552),\n",
|
||
" (Document(metadata={'source': 'Olympics FAQ', 'faq_id': 'OnllTAmy-en', 'question': 'What documentary evidence do visitors with disabilities from abroad need?', 'answer': 'Visitors will need to provide documentation to book PFR (wheelchair spaces) or PSH (easy access seating in the stands).\\n\\nThis documentation can be a European parking card or a mobility inclusion card.\\n\\nFor countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.\\n\\n\\n\\nFor countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.', 'topics': 'Spectators;Accessibility;On-site access / services', 'url': 'https://help.paris2024.org/en-gb/contents/What-documentary-evidence-do-visitors-with-disabilities-from-abroad-need-OnllTAmy', '_id': 'af36d08e-9cc3-49d2-81e3-d0e15a6825f9', '_collection_name': 'olympics-2024'}, page_content='Question: What documentary evidence do visitors with disabilities from abroad need?\\nAnswer: Visitors will need to provide documentation to book PFR (wheelchair spaces) or PSH (easy access seating in the stands).\\n\\nThis documentation can be a European parking card or a mobility inclusion card.\\n\\nFor countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.\\n\\n\\n\\nFor countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.'),\n",
|
||
" 0.8880006763829658),\n",
|
||
" (Document(metadata={'source': 'Olympics FAQ', 'faq_id': '-Bgz0H-g-en', 'question': 'Are there any specific spaces for individuals with mental/neurological/psychological disabilities?', 'answer': 'Individuals with mental, neurological, or psychological disabilities can access reserved spots for people with disabilities that are easily accessible upon presentation of a disability card or equivalent.\\n\\nAll volunteers will receive training on mental, neurological, and psychological disabilities to ensure the best possible experience and welcome.', 'topics': 'Spectators;Accessibility;On-site access / services', 'url': 'https://help.paris2024.org/en-gb/contents/Are-there-any-specific-spaces-for-individuals-with-mental-neurological-psychological-disabilities--Bgz0H-g', '_id': 'd0248842-155d-4875-a1a2-baaac8e30a27', '_collection_name': 'olympics-2024'}, page_content='Question: Are there any specific spaces for individuals with mental/neurological/psychological disabilities?\\nAnswer: Individuals with mental, neurological, or psychological disabilities can access reserved spots for people with disabilities that are easily accessible upon presentation of a disability card or equivalent.\\n\\nAll volunteers will receive training on mental, neurological, and psychological disabilities to ensure the best possible experience and welcome.'),\n",
|
||
" 0.8761557912533754)]"
|
||
]
|
||
},
|
||
"execution_count": 27,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# TODO: Try to Enhance retrieval quality\n",
|
||
"# - Find a way to Retrieve cosine similarity scores [hint: there is a method called \"similarity_search_with_relevance_scores]\n",
|
||
"# - Experiment with higher k values to return more candidate documents. But remember the more k is high the more\n",
|
||
"# - you'll have a big system prompt on the next step (generation)\n",
|
||
"similarity_scores = vector_store.similarity_search_with_relevance_scores(\n",
|
||
" query=test_query, k=3, # Return top 3 matches\n",
|
||
")\n",
|
||
"similarity_scores\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"Now that we can get similarity scores, notice that even unrelated queries return results.\n",
|
||
"Can you find a way to filter out results that are too dissimilar from the query?\n",
|
||
"\n",
|
||
"Example:\n",
|
||
"Query: \"How to become an AI Engineer?\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 26,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"[(Document(metadata={'source': 'Olympics FAQ', 'faq_id': '3KjcmsfM-en', 'question': 'What are the planned works for the area around the Grand Palais, Pont Alexandre III, and Les Invalides in preparation for the Paris 2024 Olympic and Paralympic Games?', 'answer': 'The schedule for the temporary installations of Paris 2024 in the area of the Grand Palais, Pont Alexandre III, and Les Invalides runs from April to October 2024.\\n\\n\\n\\n * Mid-April: Start of assembly on the southeast lawns of Les Invalides.\\n\\n * From mid-April to mid-June: Progressive expansion of the construction site across all the lawns of Les Invalides and on Avenue du Maréchal Gallieni; closure of Cours la Reine between Pont Alexandre III and Les Invalides from April 26 onwards and occupation of Pont Alexandre III and the nearby lower quays from May 17 onwards.\\n\\n * From September to late October: Gradual release of the Champ-de-Mars.\\n\\n\\n\\nFor more information on the assembly and dismantling of the sites:\\n\\nhttps://www.paris2024.org/fr/Montage-Demontage-Sites/%EF%BF%BC', 'topics': 'Spectators;Competition venues', 'url': 'https://help.paris2024.org/en-gb/contents/What-are-the-planned-works-for-the-area-around-the-Grand-Palais-Pont-Alexandre-III-and-Les-Invalides-in-preparation-for-the-Paris-2024-Olympic-and-Paralympic-Games-3KjcmsfM', '_id': '6c3feb47-3d53-4fcb-b9aa-463367fb5e8c', '_collection_name': 'olympics-2024'}, page_content='Question: What are the planned works for the area around the Grand Palais, Pont Alexandre III, and Les Invalides in preparation for the Paris 2024 Olympic and Paralympic Games?\\nAnswer: The schedule for the temporary installations of Paris 2024 in the area of the Grand Palais, Pont Alexandre III, and Les Invalides runs from April to October 2024.\\n\\n\\n\\n * Mid-April: Start of assembly on the southeast lawns of Les Invalides.\\n\\n * From mid-April to mid-June: Progressive expansion of the construction site across all the lawns of Les Invalides and on Avenue du Maréchal Gallieni; closure of Cours la Reine between Pont Alexandre III and Les Invalides from April 26 onwards and occupation of Pont Alexandre III and the nearby lower quays from May 17 onwards.\\n\\n * From September to late October: Gradual release of the Champ-de-Mars.\\n\\n\\n\\nFor more information on the assembly and dismantling of the sites:\\n\\nhttps://www.paris2024.org/fr/Montage-Demontage-Sites/%EF%BF%BC'),\n",
|
||
" 0.8400907480550941),\n",
|
||
" (Document(metadata={'source': 'Olympics FAQ', 'faq_id': 'v-ZwbOWG-en', 'question': 'What are the objectives of the endowment Fund?', 'answer': 'The Paris 2024 endowment Fund is a platform for social innovation through sports with three main objectives:\\n\\n * Inspire and identify projects with high potential for social innovation through sports\\n\\n * Support project leaders (sports organizations, communities, associations) in the design, implementation, and impact evaluation of these projects\\n\\n * Promote and publicize these projects to encourage their replication', 'topics': 'Other;Impact 2024', 'url': 'https://help.paris2024.org/en-gb/contents/What-are-the-objectives-of-the-endowment-Fund-v-ZwbOWG', '_id': '706433f3-76a8-4252-88a7-26b3c64eeb1f', '_collection_name': 'olympics-2024'}, page_content='Question: What are the objectives of the endowment Fund?\\nAnswer: The Paris 2024 endowment Fund is a platform for social innovation through sports with three main objectives:\\n\\n * Inspire and identify projects with high potential for social innovation through sports\\n\\n * Support project leaders (sports organizations, communities, associations) in the design, implementation, and impact evaluation of these projects\\n\\n * Promote and publicize these projects to encourage their replication'),\n",
|
||
" 0.8396205061244493),\n",
|
||
" (Document(metadata={'source': 'Olympics FAQ', 'faq_id': 'OngwUizS-en', 'question': 'Why create an account on the Génération 2024 platform?', 'answer': 'Once you have created an account on the Génération 2024 platform:\\n\\n\\n\\n * You will receive all the latest news via the quarterly newsletter\\n\\n * You will have exclusive access to educational resources such as Escape Games, films, colouring books, etc.\\n\\n\\n\\nFor more information, please visit:\\n\\nhttps://generation.paris2024.org/', 'topics': 'Education;Platform Generation 2024', 'url': 'https://help.paris2024.org/en-gb/contents/Why-create-an-account-on-the-Generation-2024-platform-OngwUizS', '_id': '9e124dde-3f7d-4419-b18d-b85ede16099c', '_collection_name': 'olympics-2024'}, page_content='Question: Why create an account on the Génération 2024 platform?\\nAnswer: Once you have created an account on the Génération 2024 platform:\\n\\n\\n\\n * You will receive all the latest news via the quarterly newsletter\\n\\n * You will have exclusive access to educational resources such as Escape Games, films, colouring books, etc.\\n\\n\\n\\nFor more information, please visit:\\n\\nhttps://generation.paris2024.org/'),\n",
|
||
" 0.8395605827531454)]"
|
||
]
|
||
},
|
||
"execution_count": 26,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# TODO: - Implement something to filter low-relevance results\n",
|
||
"# Use the following query:\n",
|
||
"query = \"Why did Napoleon Bonaparte invade Egypt ?\"\n",
|
||
"similarity_scores = vector_store.similarity_search_with_relevance_scores(\n",
|
||
" query=query, k=3, # Return top 3 matches\n",
|
||
")\n",
|
||
"similarity_scores\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"Now verify that with this new configuration, it does not filter out also relevant results\n",
|
||
"retest now with the test_query= \"How do I get disabled parking spaces?\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 29,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"[(Document(metadata={'source': 'Olympics FAQ', 'faq_id': 'OnlmLzuE-en', 'question': 'Are parking spaces reserved for individuals with disabilities available on-site?', 'answer': 'Each site will have a limited number of parking spaces reserved for wheelchair users.\\n\\nHolders of a PFR ticket will be able to access them without reservation and subject to availability, upon presentation of a European parking card, a mobility inclusion card, and a PFR ticket.\\n\\n\\n\\nFor countries not issuing either of these two cards, any other official document justifying a disability will be accepted.', 'topics': 'Spectators;Accessibility;On-site access / services', 'url': 'https://help.paris2024.org/en-gb/contents/Are-parking-spaces-reserved-for-individuals-with-disabilities-available-on-site-OnlmLzuE', '_id': 'be6db04a-2902-4ccd-bc29-d2f0d010fff4', '_collection_name': 'olympics-2024'}, page_content='Question: Are parking spaces reserved for individuals with disabilities available on-site?\\nAnswer: Each site will have a limited number of parking spaces reserved for wheelchair users.\\n\\nHolders of a PFR ticket will be able to access them without reservation and subject to availability, upon presentation of a European parking card, a mobility inclusion card, and a PFR ticket.\\n\\n\\n\\nFor countries not issuing either of these two cards, any other official document justifying a disability will be accepted.'),\n",
|
||
" 0.8989433697194552),\n",
|
||
" (Document(metadata={'source': 'Olympics FAQ', 'faq_id': 'OnllTAmy-en', 'question': 'What documentary evidence do visitors with disabilities from abroad need?', 'answer': 'Visitors will need to provide documentation to book PFR (wheelchair spaces) or PSH (easy access seating in the stands).\\n\\nThis documentation can be a European parking card or a mobility inclusion card.\\n\\nFor countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.\\n\\n\\n\\nFor countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.', 'topics': 'Spectators;Accessibility;On-site access / services', 'url': 'https://help.paris2024.org/en-gb/contents/What-documentary-evidence-do-visitors-with-disabilities-from-abroad-need-OnllTAmy', '_id': 'af36d08e-9cc3-49d2-81e3-d0e15a6825f9', '_collection_name': 'olympics-2024'}, page_content='Question: What documentary evidence do visitors with disabilities from abroad need?\\nAnswer: Visitors will need to provide documentation to book PFR (wheelchair spaces) or PSH (easy access seating in the stands).\\n\\nThis documentation can be a European parking card or a mobility inclusion card.\\n\\nFor countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.\\n\\n\\n\\nFor countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.'),\n",
|
||
" 0.8880006763829658),\n",
|
||
" (Document(metadata={'source': 'Olympics FAQ', 'faq_id': '-Bgz0H-g-en', 'question': 'Are there any specific spaces for individuals with mental/neurological/psychological disabilities?', 'answer': 'Individuals with mental, neurological, or psychological disabilities can access reserved spots for people with disabilities that are easily accessible upon presentation of a disability card or equivalent.\\n\\nAll volunteers will receive training on mental, neurological, and psychological disabilities to ensure the best possible experience and welcome.', 'topics': 'Spectators;Accessibility;On-site access / services', 'url': 'https://help.paris2024.org/en-gb/contents/Are-there-any-specific-spaces-for-individuals-with-mental-neurological-psychological-disabilities--Bgz0H-g', '_id': 'd0248842-155d-4875-a1a2-baaac8e30a27', '_collection_name': 'olympics-2024'}, page_content='Question: Are there any specific spaces for individuals with mental/neurological/psychological disabilities?\\nAnswer: Individuals with mental, neurological, or psychological disabilities can access reserved spots for people with disabilities that are easily accessible upon presentation of a disability card or equivalent.\\n\\nAll volunteers will receive training on mental, neurological, and psychological disabilities to ensure the best possible experience and welcome.'),\n",
|
||
" 0.8761557912533754)]"
|
||
]
|
||
},
|
||
"execution_count": 29,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Retrieve top 3 most relevant documents\n",
|
||
"test_query = \"How do I get disabled parking spaces?\"\n",
|
||
"retrieved_docs = vector_store.similarity_search_with_relevance_scores(\n",
|
||
" query=test_query, k=3, # Return top 3 matches\n",
|
||
")\n",
|
||
"retrieved_docs\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 🎨 Step 6: Formatting Documents for the Prompt\n",
|
||
"\n",
|
||
"We need to format our retrieved documents as a single string to pass them as context to the LLM prompt.\n",
|
||
"\n",
|
||
"### Strategy:\n",
|
||
"\n",
|
||
"Format each document as:\n",
|
||
"```\n",
|
||
"Document 1:\n",
|
||
"[content]\n",
|
||
"\n",
|
||
"Document 2:\n",
|
||
"[content]\n",
|
||
"...\n",
|
||
"```\n",
|
||
"\n",
|
||
"This makes it easy for the LLM to reference specific documents!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 32,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"🎨 Document Formatting Example:\n",
|
||
"\n",
|
||
"============================================================\n",
|
||
"Document 1:\n",
|
||
"Question: Are parking spaces reserved for individuals with disabilities available on-site?\n",
|
||
"Answer: Each site will have a limited number of parking spaces reserved for wheelchair users.\n",
|
||
"\n",
|
||
"Holders of a PFR ticket will be able to access them without reservation and subject to availability, upon presentation of a European parking card, a mobility inclusion card, and a PFR ticket.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"For countries not issuing either of these two cards, any other official document justifying a disability will be accepted.\n",
|
||
"\n",
|
||
"Document 2:\n",
|
||
"Question: What documentary evidence do visitors with disabilities from abroad need?\n",
|
||
"Answer: Visitors will need to provide documentation to book PFR (wheelchair spaces) or PSH (easy access seating in the stands).\n",
|
||
"\n",
|
||
"This documentation can be a European parking card or a mobility inclusion card.\n",
|
||
"\n",
|
||
"For countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"For countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def format_docs(docs: list[Document]) -> str:\n",
|
||
" \"\"\"Format retrieved documents for inclusion in prompt.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" docs: List of Document objects\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" Formatted string with numbered documents\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" formatted = []\n",
|
||
"\n",
|
||
" for i, doc in enumerate(docs, 1):\n",
|
||
" formatted.append(f\"Document {i}:\\n{doc.page_content}\")\n",
|
||
"\n",
|
||
" return \"\\n\\n\".join(formatted)\n",
|
||
"\n",
|
||
"\n",
|
||
"# Test the formatting\n",
|
||
"print(\"🎨 Document Formatting Example:\\n\")\n",
|
||
"print(\"=\" * 60)\n",
|
||
"\n",
|
||
"docs_to_format = [\n",
|
||
" item[0] if isinstance(item, tuple) else item for item in retrieved_docs[:2]\n",
|
||
"]\n",
|
||
"formatted_context = format_docs(docs_to_format) # Show first 2\n",
|
||
"print(formatted_context)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 34,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"🎨 Document Formatting Example:\n",
|
||
"\n",
|
||
"============================================================\n",
|
||
"Document 1 (Source: https://help.paris2024.org/en-gb/contents/Are-parking-spaces-reserved-for-individuals-with-disabilities-available-on-site-OnlmLzuE):\n",
|
||
"Question: Are parking spaces reserved for individuals with disabilities available on-site?\n",
|
||
"Answer: Each site will have a limited number of parking spaces reserved for wheelchair users.\n",
|
||
"\n",
|
||
"Holders of a PFR ticket will be able to access them without reservation and subject to availability, upon presentation of a European parking card, a mobility inclusion card, and a PFR ticket.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"For countries not issuing either of these two cards, any other official document justifying a disability will be accepted.\n",
|
||
"\n",
|
||
"Document 2 (Source: https://help.paris2024.org/en-gb/contents/What-documentary-evidence-do-visitors-with-disabilities-from-abroad-need-OnllTAmy):\n",
|
||
"Question: What documentary evidence do visitors with disabilities from abroad need?\n",
|
||
"Answer: Visitors will need to provide documentation to book PFR (wheelchair spaces) or PSH (easy access seating in the stands).\n",
|
||
"\n",
|
||
"This documentation can be a European parking card or a mobility inclusion card.\n",
|
||
"\n",
|
||
"For countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"For countries that do not issue either of these cards, any other official documentation justifying a disability will be accepted.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def format_docs_alternative(docs: list[Document]) -> str:\n",
|
||
" \"\"\"Format retrieved documents for inclusion in prompt.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" docs: List of Document objects\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" Formatted string with numbered documents\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" formatted = []\n",
|
||
"\n",
|
||
" # TODO: Optimize context format\n",
|
||
" # - Test answer-only context vs. current question+answer pairs\n",
|
||
" # Rationale: While questions improve retrieval quality, including both FAQ\n",
|
||
" # questions and user queries in the prompt may confuse the LLM and increase hallucinations\n",
|
||
" # - Include source URLs from metadata to enable citation and improve transparency\n",
|
||
" for i, doc in enumerate(docs, 1):\n",
|
||
" source_url = doc.metadata.get(\"url\", \"No URL\")\n",
|
||
" formatted.append(f\"Document {i} (Source: {source_url}):\\n{doc.page_content}\")\n",
|
||
" return \"\\n\\n\".join(formatted)\n",
|
||
"\n",
|
||
"\n",
|
||
"# Test the formatting\n",
|
||
"print(\"🎨 Document Formatting Example:\\n\")\n",
|
||
"print(\"=\" * 60)\n",
|
||
"\n",
|
||
"docs_to_format = [\n",
|
||
" item[0] if isinstance(item, tuple) else item for item in retrieved_docs[:2]\n",
|
||
"]\n",
|
||
"formatted_context = format_docs_alternative(docs_to_format) # Show first 2\n",
|
||
"print(formatted_context)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"💡Notice how there are repeated lines and multiple line breaks - we should have preprocessed this data as well! \n",
|
||
"Keep these optimization ideas in mind for improving the RAG system in the next exercise, which will be a bit more challenging! 😊"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 📝 Step 8: Crafting the Prompt\n",
|
||
"\n",
|
||
"### Why Prompts Matter:\n",
|
||
"\n",
|
||
"A well-crafted prompt is like giving clear instructions to a smart assistant. It should:\n",
|
||
"- 🎯 **Be specific:** Clear about what to do\n",
|
||
"- 📚 **Provide context:** Give relevant information\n",
|
||
"- 🚫 **Set boundaries:** What NOT to do\n",
|
||
"- ✅ **Be structured:** Easy for LLM to parse\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### Our Prompt Strategy:\n",
|
||
"\n",
|
||
"```\n",
|
||
"SYSTEM ROLE → CONTEXT (Documents) → USER QUESTION → INSTRUCTIONS\n",
|
||
"```"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 35,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"📝 Creating the RAG Prompt Template\n",
|
||
"\n",
|
||
"============================================================\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(\"📝 Creating the RAG Prompt Template\\n\")\n",
|
||
"print(\"=\" * 60)\n",
|
||
"\n",
|
||
"# Define the prompt template\n",
|
||
"prompt_template = \"\"\"You are a helpful assistant answering questions about the Olympic Games.\n",
|
||
"\n",
|
||
"Use the following context documents to answer the user's question. If the answer is not in the provided documents, say \"I don't have that information in the provided documents.\"\n",
|
||
"\n",
|
||
"Context Documents:\n",
|
||
"{context}\n",
|
||
"\n",
|
||
"User Question: {question}\n",
|
||
"\n",
|
||
"Instructions:\n",
|
||
"1. Answer based ONLY on the provided documents\n",
|
||
"2. Be specific and cite which document(s) you used\n",
|
||
"3. If information is unclear or missing, say so\n",
|
||
"4. Keep answers concise but complete\n",
|
||
"5. Use a friendly, informative tone\n",
|
||
"Answer:\"\"\"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 🤖 Step 9: Initialize the LLM\n",
|
||
"\n",
|
||
"**Mistral Large** is Mistral AI's most capable model:\n",
|
||
"\n",
|
||
"| Feature | Details | Why It Matters for RAG |\n",
|
||
"|---------|----------|----------------------|\n",
|
||
"| **Speed** | ⚡ Very fast | Quick responses to user queries |\n",
|
||
"| **Context Window** | 128K tokens | Fits many retrieved documents |\n",
|
||
"| **Max Output** | 8K tokens | Detailed answers possible |\n",
|
||
"| **Cost** | Competitive pricing | Scalable for production use |\n",
|
||
"| **Strengths** | Reasoning & instruction following | Perfect for synthesizing retrieved context |\n",
|
||
"\n",
|
||
"The ideal choice for production RAG systems! 🎯\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 36,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"MODEL_NAME = \"mistral-large-latest\"\n",
|
||
"TEMPERATURE = 0\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 37,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"🤖 Initializing the Language Model\n",
|
||
"\n",
|
||
"============================================================\n",
|
||
"response to hello: Hello! 😊 How can I assist you today? Whether you have a question, need help with something, or just want to chat, I'm here for you!\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(\"🤖 Initializing the Language Model\\n\")\n",
|
||
"print(\"=\" * 60)\n",
|
||
"# Initialize the LLM\n",
|
||
"from langchain_mistralai import ChatMistralAI\n",
|
||
"\n",
|
||
"llm = ChatMistralAI(\n",
|
||
" model=MODEL_NAME, temperature=TEMPERATURE,\n",
|
||
")\n",
|
||
"\n",
|
||
"# try an invoke\n",
|
||
"response = llm.invoke(\"hello\")\n",
|
||
"print(\"response to hello:\", response.content)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 🎯 Step 10: Building the Complete RAG Pipeline\n",
|
||
"\n",
|
||
"Now let's put it all together! We'll create a function that:\n",
|
||
"1. Takes a user question\n",
|
||
"2. Retrieves relevant documents\n",
|
||
"3. Formats them into a prompt\n",
|
||
"4. Gets an answer from the LLM\n",
|
||
"\n",
|
||
"This is the **complete RAG pipeline**! 🚀"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 38,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def make_rag_prompt(question: str, k: int = 3, score_threshold: float = 0.4) -> str:\n",
|
||
" \"\"\"Create a RAG prompt by retrieving relevant documents.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" question: User's question\n",
|
||
" k: Number of documents to retrieve\n",
|
||
" score_threshold: Minimum similarity score for retrieved documents\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" Formatted prompt string with context and question\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" # Step 1: Retrieve relevant documents\n",
|
||
" retrieved_docs = vector_store.similarity_search(\n",
|
||
" query=question, k=k, score_threshold=score_threshold,\n",
|
||
" )\n",
|
||
"\n",
|
||
" # Step 2: Format documents\n",
|
||
" context = format_docs(retrieved_docs)\n",
|
||
"\n",
|
||
" # Step 3: Fill in the prompt template\n",
|
||
" return prompt_template.format(context=context, question=question)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 39,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"You are a helpful assistant answering questions about the Olympic Games.\n",
|
||
"\n",
|
||
"Use the following context documents to answer the user's question. If the answer is not in the provided documents, say \"I don't have that information in the provided documents.\"\n",
|
||
"\n",
|
||
"Context Documents:\n",
|
||
"Document 1:\n",
|
||
"Question: I am a traveler with a disability, how can I travel by plane?\n",
|
||
"Answer: Air France offers specific services to facilitate the travel of people with reduced mobility through the SAPHIR programme (Service dedicated to Assistance for Persons with Handicaps for Information and Reservations).\n",
|
||
"\n",
|
||
"This free service is available in France as well as in 20 countries worldwide.\n",
|
||
"\n",
|
||
"To be considered, your request must reach us at least 48 hours before the departure of your first flight.\n",
|
||
"\n",
|
||
"Air France acknowledges the Sunflower symbol from the \"Hidden disabilities Sunflower®\" programme as an emblem to facilitate the identification and travel experience of its customers with invisible disabilities.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"To learn more, please visit: https://wwws.airfrance.fr/information/passagers/acheter-billet-avion-pmr-autres-handicaps\n",
|
||
"\n",
|
||
"Document 2:\n",
|
||
"Question: I have a disability, can I receive assistance at the train station?\n",
|
||
"Answer: People with disabilities and reduced mobility can benefit from free assistance at the departure and/or arrival station, from a reception point to their seat on the train.\n",
|
||
"\n",
|
||
"This service is called Assist'enGare and is available in over 1,000 stations in France. Assist'enGare can be reached:\n",
|
||
"\n",
|
||
" * Online via the reservation form 24/7:\n",
|
||
" \n",
|
||
" https://www.garesetconnexions.sncf/en/assistances-psh-pmr\n",
|
||
"\n",
|
||
" * By phone every day between 8 a.m. and 8 p.m. at 3212 and at +33 (0)9 72 72 00 92 from abroad (free service + call price).\n",
|
||
"\n",
|
||
" * Via Rogervoice, our relay center allowing deaf and hard of hearing people to communicate with our Assist'enGare teleadvisors through a translator operator. This service is available Monday to Friday from 8:30 a.m. to 9 p.m. (excluding holidays) in French Sign Language (LSF), French Spoken Complemented (LfPC), and Real-Time Speech Transcription (TTRP), and 24/7 in Text Transcription (TT).\n",
|
||
"\n",
|
||
"During peak periods, it is advisable to book this service as soon as you purchase your train ticket to ensure assistance. Without a reservation, assistance is provided subject to availability.\n",
|
||
"\n",
|
||
"Before purchasing the train ticket, it is advisable to check that assistance is available at the desired station and time.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"For more information, please visit:\n",
|
||
"\n",
|
||
"https://www.paris2024.org/en/practical-information-accessibility/\n",
|
||
"\n",
|
||
"Document 3:\n",
|
||
"Question: How will individuals with disabilities be accommodated at the competition venues?\n",
|
||
"Answer: Paris 2024 is committed to ensuring accessibility for people with disabilities at every competition venue by implementing the following measures:\n",
|
||
"\n",
|
||
" * A drop-off zone near the site entrances for PWD (People With Disabilities).\n",
|
||
"\n",
|
||
" * A parking area for PFR (People with Reduced Mobility) and a drop-off zone for IDFM shuttles for PFR.\n",
|
||
"\n",
|
||
" * Dedicated assistance available from the drop-off and parking zone.\n",
|
||
"\n",
|
||
" * A priority queue for PWD at the site entrances.\n",
|
||
"\n",
|
||
" * Dedicated reception staff for people with disabilities.\n",
|
||
"\n",
|
||
" * Mobility assistance within the site, with the option to use a wheelchair to reach the seating area.\n",
|
||
"\n",
|
||
" * A ticketing offer specifically dedicated to people with disabilities and their companions.\n",
|
||
"\n",
|
||
" * A dog relief area at each site.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"Find all the services available to PWD and PFR spectators on our dedicated page: https://www.paris2024.org/fr/infos-pratiques-accessibilite/\n",
|
||
"\n",
|
||
"User Question: what can I do if I'm disabled ?\n",
|
||
"\n",
|
||
"Instructions:\n",
|
||
"1. Answer based ONLY on the provided documents\n",
|
||
"2. Be specific and cite which document(s) you used\n",
|
||
"3. If information is unclear or missing, say so\n",
|
||
"4. Keep answers concise but complete\n",
|
||
"5. Use a friendly, informative tone\n",
|
||
"Answer:\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"question = \"what can I do if I'm disabled ?\"\n",
|
||
"print(make_rag_prompt(question, k=3, score_threshold=0.4))\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 41,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"✅ RAG Pipeline Functions Created!\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def rag_assistant(user_question: str, k: int = 3, verbose: bool = False) -> str:\n",
|
||
" \"\"\"Complete RAG pipeline: question → retrieval → generation.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" user_question: The question to answer\n",
|
||
" k: Number of documents to retrieve\n",
|
||
" verbose: Whether to print intermediate steps\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" LLM's answer string\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" if verbose:\n",
|
||
" print(\"🔄 RAG Pipeline Steps:\\n\")\n",
|
||
" print(f\"1️⃣ Query: '{user_question}'\")\n",
|
||
" print(f\"2️⃣ Retrieving top {k} documents...\")\n",
|
||
"\n",
|
||
" # Create the prompt with retrieved context\n",
|
||
" full_prompt = make_rag_prompt(user_question, k=k)\n",
|
||
"\n",
|
||
" if verbose:\n",
|
||
" print(f\"3️⃣ Prompt created ({len(full_prompt)} characters)\")\n",
|
||
" print(\"4️⃣ Generating answer with LLM...\\n\")\n",
|
||
"\n",
|
||
" # Get answer from LLM\n",
|
||
" response = llm.invoke(full_prompt)\n",
|
||
" content = response.content\n",
|
||
"\n",
|
||
" if isinstance(content, str):\n",
|
||
" return content\n",
|
||
"\n",
|
||
" return \"\\n\".join(\n",
|
||
" item if isinstance(item, str) else str(item)\n",
|
||
" for item in content\n",
|
||
" )\n",
|
||
"\n",
|
||
"\n",
|
||
"print(\"✅ RAG Pipeline Functions Created!\\n\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 🎮 Step 11: Testing the RAG System!\n",
|
||
"\n",
|
||
"Let's ask some questions and see our RAG system in action! 🚀\n",
|
||
"\n",
|
||
"### Test Questions:\n",
|
||
"\n",
|
||
"We'll test with various types of questions to see how well our system performs."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 42,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"🎮 Testing the RAG System\n",
|
||
"\n",
|
||
"======================================================================\n",
|
||
"\n",
|
||
"❓ Question 1: What can I do if i'm disabled?\n",
|
||
"\n",
|
||
"🤖 RAG Assistant's Answer:\n",
|
||
"\n",
|
||
"Here’s what you can do if you’re a person with a disability (PWD) or have reduced mobility (PFR) for the Paris 2024 Olympic and Paralympic Games, based on the provided documents:\n",
|
||
"\n",
|
||
"### **At Competition Venues (Document 1)**\n",
|
||
"You’ll have access to:\n",
|
||
"- **Drop-off zones** near entrances for PWD.\n",
|
||
"- **Dedicated parking** for PFR and shuttle drop-off zones.\n",
|
||
"- **Priority queues** at entrances.\n",
|
||
"- **Assistance staff** to help from arrival to seating.\n",
|
||
"- **Wheelchair mobility support** within venues.\n",
|
||
"- **Special ticketing offers** for PWD and companions.\n",
|
||
"- **Dog relief areas** at each site.\n",
|
||
"\n",
|
||
"🔗 More details: [Paris 2024 Accessibility Page](https://www.paris2024.org/fr/infos-pratiques-accessibilite/)\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### **Train Travel (Document 2)**\n",
|
||
"- **Free assistance** (Assist’enGare) at over 1,000 stations in France, from arrival to your train seat.\n",
|
||
"- **Booking options**:\n",
|
||
" - Online form (24/7): [SNCF Assistance](https://www.garesetconnexions.sncf/en/assistances-psh-pmr)\n",
|
||
" - Phone: 3212 (France) or +33 (0)9 72 72 00 92 (abroad).\n",
|
||
" - **For deaf/hard of hearing**: Rogervoice relay service (LSF, LfPC, TTRP, or text).\n",
|
||
"- **Book early** during peak periods to guarantee assistance.\n",
|
||
"\n",
|
||
"🔗 More info: [Paris 2024 Accessibility](https://www.paris2024.org/en/practical-information-accessibility/)\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### **Air Travel (Document 3)**\n",
|
||
"- **Air France’s SAPHIR program** offers free assistance for PWD/PFR.\n",
|
||
" - Request at least **48 hours before departure**.\n",
|
||
" - Recognizes the **Sunflower symbol** for invisible disabilities.\n",
|
||
"🔗 Details: [Air France Accessibility](https://wwws.airfrance.fr/information/passagers/acheter-billet-avion-pmr-autres-handicaps)\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### **Tickets (Document 5)**\n",
|
||
"- **Special seating** for PWD, including wheelchair-accessible spots.\n",
|
||
"- Purchase through the [official ticketing site](https://tickets.paris2024.org).\n",
|
||
"🔗 FAQ: [Accessible Ticket Categories](https://tickets.paris2024.org/faq/en_en/category/accessibility/what-categories-are-available-for-people-with-disabilities)\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"======================================================================\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(\"🎮 Testing the RAG System\\n\")\n",
|
||
"print(\"=\" * 70)\n",
|
||
"\n",
|
||
"# Test Question 1\n",
|
||
"question1 = \"What can I do if i'm disabled?\"\n",
|
||
"\n",
|
||
"print(f\"\\n❓ Question 1: {question1}\\n\")\n",
|
||
"print(\"🤖 RAG Assistant's Answer:\\n\")\n",
|
||
"print(rag_assistant(question1, verbose=False, k=5))\n",
|
||
"print(\"\\n\" + \"=\" * 70)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 🎨 Step 12: Interactive Q&A Session with Gradio\n",
|
||
"\n",
|
||
"Now it's your turn! Ask your own questions about the Olympics!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 43,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"* Running on local URL: http://127.0.0.1:7860\n",
|
||
"* To create a public link, set `share=True` in `launch()`.\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><iframe src=\"http://127.0.0.1:7860/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
|
||
],
|
||
"text/plain": [
|
||
"<IPython.core.display.HTML object>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": []
|
||
},
|
||
"execution_count": 43,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"import gradio as gr\n",
|
||
"\n",
|
||
"K = 5\n",
|
||
"\n",
|
||
"\n",
|
||
"def rag_assistant_response(message: str, _history: list) -> str:\n",
|
||
" \"\"\"Gradio response function for RAG assistant.\"\"\"\n",
|
||
" return rag_assistant(message, k=K, verbose=False)\n",
|
||
"\n",
|
||
"\n",
|
||
"gr.ChatInterface(fn=rag_assistant_response).launch()\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 📊 Step 14: RAG System Analysis\n",
|
||
"\n",
|
||
"Let's analyze our RAG system's performance and characteristics."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 💡 Step 15: Understanding RAG Performance\n",
|
||
"\n",
|
||
"### 🎯 What Makes a Good RAG System?\n",
|
||
"\n",
|
||
"| Metric | Good | Excellent | Why It Matters |\n",
|
||
"|--------|------|-----------|----------------|\n",
|
||
"| **Retrieval Accuracy** | 70-80% | 90%+ | Finding the right documents |\n",
|
||
"| **Answer Quality** | Relevant | Precise & cited | Useful to users |\n",
|
||
"| **Response Time** | <2s | <1s | User experience |\n",
|
||
"| **Hallucination Rate** | <10% | <5% | Trustworthiness |\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### 🔧 Ways to Improve RAG:\n",
|
||
"\n",
|
||
"1. **Better Chunking**\n",
|
||
" - Optimal chunk size (100-500 tokens)\n",
|
||
" - Semantic chunking (not just character count)\n",
|
||
" - Overlap between chunks\n",
|
||
"\n",
|
||
"2. **Improved Retrieval**\n",
|
||
" - Hybrid search (keyword + semantic)\n",
|
||
" - Reranking retrieved documents\n",
|
||
" - Query expansion/reformulation\n",
|
||
"\n",
|
||
"3. **Better Embeddings**\n",
|
||
" - Domain-specific embedding models\n",
|
||
" - Fine-tuned embeddings\n",
|
||
" - Multi-vector approaches\n",
|
||
"\n",
|
||
"4. **Prompt Engineering**\n",
|
||
" - Clear instructions\n",
|
||
" - Few-shot examples\n",
|
||
" - Chain-of-thought reasoning\n",
|
||
"\n",
|
||
"5. **Advanced Techniques**\n",
|
||
" - Self-querying retrieval\n",
|
||
" - Hypothetical document embeddings (HyDE)\n",
|
||
" - Multi-query retrieval"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 🎉 Congratulations!\n",
|
||
"\n",
|
||
"You've successfully built a complete RAG system! Here's what you accomplished:\n",
|
||
"\n",
|
||
"## 📚 Skills Acquired:\n",
|
||
"\n",
|
||
"✅ **Data Preparation**\n",
|
||
" - Loading and structuring documents\n",
|
||
" - Creating LangChain Document objects\n",
|
||
" - Preprocessing for RAG\n",
|
||
"\n",
|
||
"✅ **Embeddings & Vector Search**\n",
|
||
" - Converting text to embeddings\n",
|
||
" - Building Qdrant vector databases\n",
|
||
" - Performing similarity search\n",
|
||
"\n",
|
||
"✅ **Prompt Engineering**\n",
|
||
" - Crafting effective RAG prompts\n",
|
||
" - Context formatting\n",
|
||
" - Instruction design\n",
|
||
"\n",
|
||
"✅ **LLM Integration**\n",
|
||
" - Using ChatOpenAI API\n",
|
||
" - Model configuration\n",
|
||
" - Response generation\n",
|
||
"\n",
|
||
"✅ **System Design**\n",
|
||
" - Building end-to-end pipelines\n",
|
||
" - Performance optimization\n",
|
||
"\n",
|
||
"---"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 🏋️ Exercises\n",
|
||
"\n",
|
||
"Time to practice! The exercises below go from basic retrieval improvements to advanced RAG patterns. Each exercise builds on the pipeline you built in this notebook.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 🔍 Exercise 1: Score-Threshold Filtering\n",
|
||
"\n",
|
||
"Right now the retriever always returns results, even when the query is completely unrelated to the Olympics. Implement a filtered retrieval that **rejects documents below a similarity threshold**.\n",
|
||
"\n",
|
||
"**Requirements:**\n",
|
||
"- Use `similarity_search_with_relevance_scores()` to get scores alongside documents\n",
|
||
"- Filter out any document whose score is below `score_threshold`\n",
|
||
"- Return an empty list (and a friendly message) when nothing passes the threshold\n",
|
||
"- Test with a clearly relevant query and a clearly irrelevant one\n",
|
||
"\n",
|
||
"**Test queries:**\n",
|
||
"- `\"How do I get tickets for the swimming finals?\"` → should return results\n",
|
||
"- `\"Who invented the telephone?\"` → should return nothing\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 44,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Query: 'How do I get tickets for the swimming finals?'\n",
|
||
"Results: 5 document(s) found\n",
|
||
"\n",
|
||
"Query: 'Who invented the telephone?'\n",
|
||
"⚠️ 5 document(s) returned (lower your threshold?)\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def filtered_retrieval(query: str, k: int = 5, score_threshold: float = 0.6) -> list[Document]:\n",
|
||
" \"\"\"Retrieve documents only if their similarity score exceeds the threshold.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" query: The user question\n",
|
||
" k: Maximum number of documents to retrieve\n",
|
||
" score_threshold: Minimum cosine similarity score (0 to 1)\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" List of relevant Document objects (empty if nothing passes the threshold)\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" # TODO: use vector_store.similarity_search_with_relevance_scores()\n",
|
||
" # to get (Document, score) tuples, then filter by score_threshold\n",
|
||
"\n",
|
||
" results_with_scores = vector_store.similarity_search_with_relevance_scores(\n",
|
||
" query=query, k=k,\n",
|
||
" )\n",
|
||
"\n",
|
||
" # Filter and extract only the documents that pass the threshold\n",
|
||
" return [\n",
|
||
" doc for doc, score in results_with_scores if score >= score_threshold\n",
|
||
" ]\n",
|
||
"\n",
|
||
"\n",
|
||
"# Test 1: Relevant query — should return results\n",
|
||
"relevant_query = \"How do I get tickets for the swimming finals?\"\n",
|
||
"docs = filtered_retrieval(relevant_query)\n",
|
||
"print(f\"Query: '{relevant_query}'\")\n",
|
||
"print(f\"Results: {len(docs)} document(s) found\\n\")\n",
|
||
"\n",
|
||
"# Test 2: Irrelevant query — should return nothing\n",
|
||
"irrelevant_query = \"Who invented the telephone?\"\n",
|
||
"docs = filtered_retrieval(irrelevant_query)\n",
|
||
"print(f\"Query: '{irrelevant_query}'\")\n",
|
||
"if not docs:\n",
|
||
" print(\"✅ Correctly filtered out — no relevant documents found.\")\n",
|
||
"else:\n",
|
||
" print(f\"⚠️ {len(docs)} document(s) returned (lower your threshold?)\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 🎨 Exercise 2: Better Document Formatter with Citations\n",
|
||
"\n",
|
||
"The current `format_docs()` includes full Q&A text. Your task is to complete `format_docs_alternative()` so that it:\n",
|
||
"\n",
|
||
"1. Includes **only the answer** (not the question) to avoid confusing the LLM\n",
|
||
"2. Adds a **source URL** from the document metadata for citations\n",
|
||
"3. Formats each block clearly so the LLM can reference it\n",
|
||
"\n",
|
||
"**Expected output format:**\n",
|
||
"```\n",
|
||
"Document 1:\n",
|
||
"[answer text]\n",
|
||
"Source: https://...\n",
|
||
"\n",
|
||
"Document 2:\n",
|
||
"[answer text]\n",
|
||
"Source: https://...\n",
|
||
"```\n",
|
||
"\n",
|
||
"Then plug your new formatter into `rag_assistant()` and compare answers!\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 45,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== Original format_docs ===\n",
|
||
"Document 1:\n",
|
||
"Question: I have a disability, can I receive assistance at the train station?\n",
|
||
"Answer: People with disabilities and reduced mobility can benefit from free assistance at the departure and/or arrival station, from a reception point to their seat on the train.\n",
|
||
"\n",
|
||
"This service is called Assist'enGare and is available in over 1,000 stations in France. Assist'enGare can be reached:\n",
|
||
"\n",
|
||
" * Online via the reservation form 24/7:\n",
|
||
" \n",
|
||
" https://www.garesetconnexions.sncf/en/assistances-psh-pmr\n",
|
||
"\n",
|
||
" * By phone every day between 8 a.m. and 8 p.m. at 3212 and at +33 (0)9 72 72 00 92 from abroad (free service + call price).\n",
|
||
"\n",
|
||
" * Via Rogervoice, our relay center allowing deaf and hard of hearing people to communicate with our Assist'enGare teleadvisors through a translator operator. This service is available Monday to Friday from 8:30 a.m. to 9 p.m. (excluding holidays) in French Sign Language (LSF), French Spoken Complemented (LfPC), and Real-Time Speech Transcription (TTRP), and 24/7 in Text Transcription (TT).\n",
|
||
"\n",
|
||
"During peak periods, it is advisable to book this service as soon as you purchase your train ticket to ensure assistance. Without a reservation, assistance is provided subject to availability.\n",
|
||
"\n",
|
||
"Before purchasing the train ticket, it is advisable to check that assistance is available at the desired station and time.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"For more information, please visit:\n",
|
||
"\n",
|
||
"https://www.paris2024.org/en/practical-information-accessibility/\n",
|
||
"\n",
|
||
"Document 2:\n",
|
||
"Question: How will individuals with disabilities be accommodated at the competition venues?\n",
|
||
"Answer: Paris 2024 is committed to ensuring accessibility for people with disabilities at every competition venue by implementing the following measures:\n",
|
||
"\n",
|
||
" * A drop-off zone near the site entrances for PWD (People With Disabilities).\n",
|
||
"\n",
|
||
" * A parking area for PFR (People with Reduced Mobility) and a drop-off zone for IDFM shuttles for PFR.\n",
|
||
"\n",
|
||
" * Dedicated assistance available from the drop-off and parking zone.\n",
|
||
"\n",
|
||
" * A priority queue for PWD at the site entrances.\n",
|
||
"\n",
|
||
" * Dedicated reception staff for people with disabilities.\n",
|
||
"\n",
|
||
" * Mobility assistance within the site, with the option to use a wheelchair to reach the seating area.\n",
|
||
"\n",
|
||
" * A ticketing offer specifically dedicated to people with disabilities and their companions.\n",
|
||
"\n",
|
||
" * A dog relief area at each site.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"Find all the services available to PWD and PFR spectators on our dedicated page: https://www.paris2024.org/fr/infos-pratiques-accessibilite/\n",
|
||
"\n",
|
||
"=== New format_docs_with_citations ===\n",
|
||
"Document 1:\n",
|
||
"People with disabilities and reduced mobility can benefit from free assistance at the departure and/or arrival station, from a reception point to their seat on the train.\n",
|
||
"\n",
|
||
"This service is called Assist'enGare and is available in over 1,000 stations in France. Assist'enGare can be reached:\n",
|
||
"\n",
|
||
" * Online via the reservation form 24/7:\n",
|
||
" \n",
|
||
" https://www.garesetconnexions.sncf/en/assistances-psh-pmr\n",
|
||
"\n",
|
||
" * By phone every day between 8 a.m. and 8 p.m. at 3212 and at +33 (0)9 72 72 00 92 from abroad (free service + call price).\n",
|
||
"\n",
|
||
" * Via Rogervoice, our relay center allowing deaf and hard of hearing people to communicate with our Assist'enGare teleadvisors through a translator operator. This service is available Monday to Friday from 8:30 a.m. to 9 p.m. (excluding holidays) in French Sign Language (LSF), French Spoken Complemented (LfPC), and Real-Time Speech Transcription (TTRP), and 24/7 in Text Transcription (TT).\n",
|
||
"\n",
|
||
"During peak periods, it is advisable to book this service as soon as you purchase your train ticket to ensure assistance. Without a reservation, assistance is provided subject to availability.\n",
|
||
"\n",
|
||
"Before purchasing the train ticket, it is advisable to check that assistance is available at the desired station and time.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"For more information, please visit:\n",
|
||
"\n",
|
||
"https://www.paris2024.org/en/practical-information-accessibility/\n",
|
||
"Source: https://help.paris2024.org/en-gb/contents/I-have-a-disability-can-I-receive-assistance-at-the-train-station-8_44nFrI\n",
|
||
"\n",
|
||
"Document 2:\n",
|
||
"Paris 2024 is committed to ensuring accessibility for people with disabilities at every competition venue by implementing the following measures:\n",
|
||
"\n",
|
||
" * A drop-off zone near the site entrances for PWD (People With Disabilities).\n",
|
||
"\n",
|
||
" * A parking area for PFR (People with Reduced Mobility) and a drop-off zone for IDFM shuttles for PFR.\n",
|
||
"\n",
|
||
" * Dedicated assistance available from the drop-off and parking zone.\n",
|
||
"\n",
|
||
" * A priority queue for PWD at the site entrances.\n",
|
||
"\n",
|
||
" * Dedicated reception staff for people with disabilities.\n",
|
||
"\n",
|
||
" * Mobility assistance within the site, with the option to use a wheelchair to reach the seating area.\n",
|
||
"\n",
|
||
" * A ticketing offer specifically dedicated to people with disabilities and their companions.\n",
|
||
"\n",
|
||
" * A dog relief area at each site.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"Find all the services available to PWD and PFR spectators on our dedicated page: https://www.paris2024.org/fr/infos-pratiques-accessibilite/\n",
|
||
"Source: https://help.paris2024.org/en-gb/contents/How-will-individuals-with-disabilities-be-accommodated-at-the-competition-venues-OnjlYofa\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def format_docs_with_citations(docs: list[Document]) -> str:\n",
|
||
" \"\"\"Format retrieved documents using answers only + source URLs.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" docs: List of Document objects (each has metadata[\"answer\"] and metadata[\"url\"])\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" Formatted string ready for the prompt\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" formatted = []\n",
|
||
"\n",
|
||
" for i, doc in enumerate(docs, 1):\n",
|
||
" # TODO: Extract only the answer from metadata (not the full page_content)\n",
|
||
" answer = doc.metadata[\"answer\"]\n",
|
||
"\n",
|
||
" # TODO: Extract the source URL from metadata\n",
|
||
" url = doc.metadata[\"url\"]\n",
|
||
"\n",
|
||
" # TODO: Build the formatted block\n",
|
||
" block = f\"Document {i}:\\n{answer}\\nSource: {url}\"\n",
|
||
"\n",
|
||
" formatted.append(block)\n",
|
||
"\n",
|
||
" return \"\\n\\n\".join(formatted)\n",
|
||
"\n",
|
||
"\n",
|
||
"# Test: compare both formatters on the same retrieved docs\n",
|
||
"test_q = \"What can I do if I'm disabled?\"\n",
|
||
"docs = vector_store.similarity_search(test_q, k=2)\n",
|
||
"\n",
|
||
"print(\"=== Original format_docs ===\")\n",
|
||
"print(format_docs(docs))\n",
|
||
"print(\"\\n=== New format_docs_with_citations ===\")\n",
|
||
"print(format_docs_with_citations(docs))\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 🔄 Exercise 3: Multi-Query Retrieval\n",
|
||
"\n",
|
||
"A single embedding can miss relevant documents phrased differently. **Multi-query retrieval** asks the LLM to rephrase the user question into N variations, runs each variation through the vector store, then **deduplicates** the results.\n",
|
||
"\n",
|
||
"**Requirements:**\n",
|
||
"- Use the LLM to generate `n_variants` alternative queries from the original question\n",
|
||
"- Run `vector_store.similarity_search()` for each variant\n",
|
||
"- Deduplicate results by `faq_id` from metadata (to avoid showing the same doc twice)\n",
|
||
"- Return the merged, deduplicated list of documents\n",
|
||
"\n",
|
||
"**Why it helps:** Different phrasings activate different regions of the embedding space, increasing recall.\n",
|
||
"\n",
|
||
"**Test with:** `\"accessibility services for people with mobility issues\"`\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 47,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Original query: 'accessibility services for people with mobility issues'\n",
|
||
"\n",
|
||
"Standard retrieval: 3 docs\n",
|
||
"Multi-query retrieval: 0 unique docs\n",
|
||
"\n",
|
||
"📄 Unique documents found:\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def generate_query_variants(question: str, n_variants: int = 3) -> list[str]:\n",
|
||
" \"\"\"Use the LLM to generate alternative phrasings of the question.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" question: Original user question\n",
|
||
" n_variants: Number of alternative queries to generate\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" List of alternative query strings (NOT including the original)\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" prompt = f\"\"\"Generate {n_variants} different ways to ask the following question.\n",
|
||
"Return ONLY the questions, one per line, with no numbering or extra text.\n",
|
||
"\n",
|
||
"Original question: {question}\"\"\"\n",
|
||
"\n",
|
||
" # TODO: call llm.invoke(prompt) and parse the response into a list of strings\n",
|
||
" response = llm.invoke(prompt)\n",
|
||
" return response.content.strip().split(\"\\n\")\n",
|
||
"\n",
|
||
"\n",
|
||
"def multi_query_retrieval(question: str, k: int = 3, n_variants: int = 3) -> list[Document]:\n",
|
||
" \"\"\"Retrieve documents using multiple query variants and deduplicate results.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" question: Original user question\n",
|
||
" k: Documents to retrieve per query variant\n",
|
||
" n_variants: Number of alternative queries to generate\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" Deduplicated list of relevant Document objects\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" all_queries = [question, *generate_query_variants(question, n_variants)]\n",
|
||
"\n",
|
||
" seen_ids = set()\n",
|
||
" unique_docs = []\n",
|
||
"\n",
|
||
" for query in all_queries:\n",
|
||
" docs = vector_store.similarity_search(query, k=k)\n",
|
||
" for doc in docs:\n",
|
||
" faq_id = doc.metadata.get(\"faq_id\")\n",
|
||
" # TODO: only add the doc if its faq_id has not been seen yet\n",
|
||
" if faq_id not in seen_ids:\n",
|
||
" ... # add to seen_ids and unique_docs\n",
|
||
"\n",
|
||
" return unique_docs\n",
|
||
"\n",
|
||
"\n",
|
||
"# Test it!\n",
|
||
"test_question = \"accessibility services for people with mobility issues\"\n",
|
||
"print(f\"Original query: '{test_question}'\\n\")\n",
|
||
"\n",
|
||
"# Standard retrieval\n",
|
||
"standard_docs = vector_store.similarity_search(test_question, k=3)\n",
|
||
"print(f\"Standard retrieval: {len(standard_docs)} docs\")\n",
|
||
"\n",
|
||
"# Multi-query retrieval\n",
|
||
"multi_docs = multi_query_retrieval(test_question, k=3, n_variants=3)\n",
|
||
"print(f\"Multi-query retrieval: {len(multi_docs)} unique docs\")\n",
|
||
"print(\"\\n📄 Unique documents found:\")\n",
|
||
"for i, doc in enumerate(multi_docs, 1):\n",
|
||
" print(f\" {i}. {doc.metadata['question'][:80]}\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 💬 Exercise 4: Conversational RAG with Memory\n",
|
||
"\n",
|
||
"The current `rag_assistant()` function is **stateless** — each call is independent. Build a `ConversationalRAG` class that remembers previous turns and uses the full conversation history when generating answers.\n",
|
||
"\n",
|
||
"**Requirements:**\n",
|
||
"- Store the conversation as a list of `(question, answer)` tuples\n",
|
||
"- Before retrieving, **rewrite the current question** using the LLM to make it standalone (e.g. resolve pronouns like \"it\", \"they\", \"there\")\n",
|
||
"- Append the conversation history to the prompt so the LLM has context from previous turns\n",
|
||
"- Add a `reset()` method to clear the history\n",
|
||
"\n",
|
||
"**Example interaction:**\n",
|
||
"```\n",
|
||
"User: What are the ticket prices?\n",
|
||
"Bot: [answer about ticket prices]\n",
|
||
"User: Can I get a refund for them? ← \"them\" needs to be resolved\n",
|
||
"Bot: [answer about ticket refunds, using history]\n",
|
||
"```\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 48,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"🎯 Conversational RAG Demo\n",
|
||
"============================================================\n",
|
||
"\n",
|
||
"👤 What kind of transport options are available to get to the venues?\n",
|
||
"🤖 Based on the provided context documents, here are the transport options available to get to the Paris 2024 Olympic and Paralympic competition venues:\n",
|
||
"\n",
|
||
"1. **Public Transport** – All venues are accessible via the **Île-de-France Mobilités network**, which includes:\n",
|
||
" - Metro\n",
|
||
" - RER (regional expres...\n",
|
||
"\n",
|
||
"👤 Is it free?\n",
|
||
"🔄 Rewritten: 'Will public transportation to the Paris 2024 Olympic and Paralympic venues be free for attendees?'\n",
|
||
"🤖 Based on the provided context documents, **public transportation to the Paris 2024 Olympic and Paralympic venues will not be free for attendees**. However, there will be an **adapted fare package** available for spectators during the Games.\n",
|
||
"\n",
|
||
"Here’s what we know:\n",
|
||
"- The **Île-de-France Mobilités netwo...\n",
|
||
"\n",
|
||
"👤 What about for disabled people?\n",
|
||
"🔄 Rewritten: 'What specific transport accommodations and accessibility options will be available for disabled attendees traveling to the Paris 2024 Olympic and Paralympic venues?'\n",
|
||
"🤖 Based on the provided context documents, here are the **specific transport accommodations and accessibility options** available for disabled attendees (PWD – People With Disabilities and PFR – People with Reduced Mobility) traveling to the **Paris 2024 Olympic and Paralympic venues**:\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### **1....\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"class ConversationalRAG:\n",
|
||
" \"\"\"A stateful RAG assistant that remembers previous conversation turns.\"\"\"\n",
|
||
"\n",
|
||
" def __init__(self, k: int = 3) -> None:\n",
|
||
" \"\"\"Initialize the assistant with an optional k for retrieval.\"\"\"\n",
|
||
" self.k = k\n",
|
||
" self.history: list[tuple[str, str]] = [] # list of (question, answer) pairs\n",
|
||
"\n",
|
||
" def reset(self) -> None:\n",
|
||
" \"\"\"Clear the conversation history.\"\"\"\n",
|
||
" self.history = []\n",
|
||
"\n",
|
||
" def _make_standalone_question(self, question: str) -> str:\n",
|
||
" \"\"\"Rewrite the question to be self-contained, resolving any references.\n",
|
||
"\n",
|
||
" to previous turns (e.g. pronouns, 'there', 'them', 'those').\n",
|
||
" If no history exists, return the question unchanged.\n",
|
||
" \"\"\"\n",
|
||
" if not self.history:\n",
|
||
" return question\n",
|
||
"\n",
|
||
" # TODO: Build a prompt that includes the conversation history\n",
|
||
" # and asks the LLM to rewrite the question to be standalone\n",
|
||
" history_text = \"\\n\".join(\n",
|
||
" [f\"User: {q}\\nAssistant: {a}\" for q, a in self.history[-3:]],\n",
|
||
" )\n",
|
||
" rewrite_prompt = f\"\"\"Given the conversation below, rewrite the follow-up question to be a fully standalone question that can be understood without the conversation context. Return ONLY the rewritten question, nothing else.\n",
|
||
"\n",
|
||
"Conversation:\n",
|
||
"{history_text}\n",
|
||
"\n",
|
||
"Follow-up question: {question}\n",
|
||
"Standalone question:\"\"\"\n",
|
||
"\n",
|
||
" # TODO: invoke the llm and return the rewritten question\n",
|
||
" response = llm.invoke(rewrite_prompt)\n",
|
||
" return response.content.strip()\n",
|
||
"\n",
|
||
" def _build_prompt(self, standalone_question: str, context: str) -> str:\n",
|
||
" \"\"\"Build the full prompt including conversation history.\"\"\"\n",
|
||
" # TODO: Format recent history as a readable exchange\n",
|
||
" history_text = \"\"\n",
|
||
" if self.history:\n",
|
||
" history_lines = []\n",
|
||
" for q, a in self.history[-3:]:\n",
|
||
" history_lines.append(f\"User: {q}\\nAssistant: {a}\")\n",
|
||
" history_text = \"\\n\\n\".join(history_lines)\n",
|
||
"\n",
|
||
" return f\"\"\"You are a helpful assistant answering questions about the Olympic Games.\n",
|
||
"\n",
|
||
"Use the context documents below to answer the question. If the answer is not in the documents, say so.\n",
|
||
"\n",
|
||
"Context Documents:\n",
|
||
"{context}\n",
|
||
"\n",
|
||
"{\"Previous conversation:\" + chr(10) + history_text if history_text else \"\"}\n",
|
||
"\n",
|
||
"User: {standalone_question}\n",
|
||
"Assistant:\"\"\"\n",
|
||
"\n",
|
||
" def chat(self, question: str, verbose: bool = False) -> str:\n",
|
||
" \"\"\"Answer a question, using conversation history for context.\"\"\"\n",
|
||
" # Step 1: Rewrite the question to be standalone\n",
|
||
" standalone_q = self._make_standalone_question(question)\n",
|
||
" if verbose and standalone_q != question:\n",
|
||
" print(f\"🔄 Rewritten: '{standalone_q}'\")\n",
|
||
"\n",
|
||
" # TODO:\n",
|
||
" # Step 2: Retrieve relevant documents using the standalone question\n",
|
||
" docs = vector_store.similarity_search(standalone_q, k=self.k)\n",
|
||
"\n",
|
||
" # Step 3: Format documents into context\n",
|
||
" context = \"\\n\\n\".join([doc.page_content for doc in docs])\n",
|
||
"\n",
|
||
" # Step 4: Build the prompt\n",
|
||
" full_prompt = self._build_prompt(standalone_q, context)\n",
|
||
"\n",
|
||
" # Step 5: Invoke the LLM\n",
|
||
" response = llm.invoke(full_prompt)\n",
|
||
"\n",
|
||
" # Step 6: Store the turn in history\n",
|
||
" answer = response.content if isinstance(response.content, str) else str(response.content)\n",
|
||
" self.history.append((question, answer))\n",
|
||
"\n",
|
||
" return answer\n",
|
||
"\n",
|
||
"\n",
|
||
"# Test it!\n",
|
||
"bot = ConversationalRAG(k=3)\n",
|
||
"\n",
|
||
"conversation = [\n",
|
||
" (\"What kind of transport options are available to get to the venues?\", False),\n",
|
||
" (\"Is it free?\", True), # \"it\" refers to transport\n",
|
||
" (\"What about for disabled people?\", True), # \"disabled people\" + transport context\n",
|
||
"]\n",
|
||
"\n",
|
||
"print(\"🎯 Conversational RAG Demo\\n\" + \"=\" * 60)\n",
|
||
"for question, verbose in conversation:\n",
|
||
" print(f\"\\n👤 {question}\")\n",
|
||
" answer = bot.chat(question, verbose=verbose)\n",
|
||
" print(f\"🤖 {answer[:300]}...\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 📊 Exercise 5: RAG Evaluation with LLM-as-Judge\n",
|
||
"\n",
|
||
"A RAG system is only as good as you can measure it. Implement a simple **LLM-as-Judge** evaluator: given a question, a retrieved context, and a generated answer, ask the LLM to score the answer on two criteria.\n",
|
||
"\n",
|
||
"**Requirements:**\n",
|
||
"Implement `evaluate_rag_answer()` which returns a dict with:\n",
|
||
"- `faithfulness` (0–5): Is the answer grounded in the provided context only?\n",
|
||
"- `relevance` (0–5): Does the answer actually address the question?\n",
|
||
"- `feedback`: A short explanation of the scores\n",
|
||
"\n",
|
||
"Then run the evaluator on at least 3 questions — one clearly relevant, one where the answer was made up, and one where the retrieval was bad.\n",
|
||
"\n",
|
||
"**Hint:** Ask the LLM to respond in JSON format for easy parsing.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 50,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"📊 RAG Evaluation Results\n",
|
||
"============================================================\n",
|
||
"\n",
|
||
"✅ Good case: \"What can I do if I'm disabled?\"\n",
|
||
" Faithfulness : 5/5\n",
|
||
" Relevance : 5/5\n",
|
||
" Feedback : The answer accurately and comprehensively summarizes all key points from the retrieved context, providing clear, actionable information for disabled individuals across train stations, Olympic venues, and air travel. No inaccuracies or omissions were detected.\n",
|
||
"\n",
|
||
"❌ Hallucination case: \"What is the price of a gold medal?\"\n",
|
||
" Faithfulness : 0/5\n",
|
||
" Relevance : 1/5\n",
|
||
" Feedback : The generated answer is entirely fabricated—the retrieved context contains no information about the price or material composition of a gold medal. The question is relevant to the Olympics, but the answer is unsupported by the provided context.\n",
|
||
"\n",
|
||
"⚠️ Off-topic retrieval: \"How do I become an Olympic athlete?\"\n",
|
||
" Faithfulness : 4/5\n",
|
||
" Relevance : 3/5\n",
|
||
" Feedback : The answer is mostly faithful to the retrieved context, particularly Document 2, which explains the qualification process. However, it does not provide a direct or comprehensive step-by-step guide on becoming an Olympic athlete, as the context lacks specific details. The relevance is moderate because while the answer addresses the question, it relies on external suggestions rather than concrete information from the provided documents.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"import json\n",
|
||
"\n",
|
||
"\n",
|
||
"def evaluate_rag_answer(question: str, context: str, answer: str) -> dict:\n",
|
||
" \"\"\"Use the LLM as a judge to evaluate the quality of a RAG answer.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" question: The original user question\n",
|
||
" context: The retrieved documents (as a formatted string)\n",
|
||
" answer: The generated answer to evaluate\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" dict with keys: \"faithfulness\" (0-5), \"relevance\" (0-5), \"feedback\" (str)\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" eval_prompt = f\"\"\"You are an expert evaluator for RAG (Retrieval-Augmented Generation) systems.\n",
|
||
"\n",
|
||
"Evaluate the answer below on two criteria. Return your evaluation as valid JSON only, with no extra text.\n",
|
||
"\n",
|
||
"QUESTION: {question}\n",
|
||
"\n",
|
||
"RETRIEVED CONTEXT:\n",
|
||
"{context}\n",
|
||
"\n",
|
||
"GENERATED ANSWER:\n",
|
||
"{answer}\n",
|
||
"\n",
|
||
"Evaluation criteria:\n",
|
||
"- faithfulness (0-5): Is every claim in the answer supported by the context? 5 = fully grounded, 0 = completely hallucinated\n",
|
||
"- relevance (0-5): Does the answer address the question? 5 = perfect, 0 = completely off-topic\n",
|
||
"- feedback: One or two sentences explaining the scores\n",
|
||
"\n",
|
||
"Respond with this exact JSON structure:\n",
|
||
"{{\"faithfulness\": <int>, \"relevance\": <int>, \"feedback\": \"<string>\"}}\"\"\"\n",
|
||
"\n",
|
||
" # TODO: invoke the LLM with the eval_prompt\n",
|
||
" response = llm.invoke(eval_prompt)\n",
|
||
"\n",
|
||
" # TODO: parse the JSON response and return the dict\n",
|
||
" response_text = response.content if isinstance(response.content, str) else str(response.content)\n",
|
||
" return json.loads(response_text.strip())\n",
|
||
"\n",
|
||
"\n",
|
||
"# Build a test harness\n",
|
||
"test_cases = [\n",
|
||
" {\n",
|
||
" \"label\": \"✅ Good case\",\n",
|
||
" \"question\": \"What can I do if I'm disabled?\",\n",
|
||
" \"use_rag\": True, # use the actual RAG pipeline\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"label\": \"❌ Hallucination case\",\n",
|
||
" \"question\": \"What is the price of a gold medal?\",\n",
|
||
" \"use_rag\": False,\n",
|
||
" \"fake_answer\": \"Each gold medal costs exactly $5,000 and is made of pure 24-karat gold.\",\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"label\": \"⚠️ Off-topic retrieval\",\n",
|
||
" \"question\": \"How do I become an Olympic athlete?\",\n",
|
||
" \"use_rag\": True,\n",
|
||
" },\n",
|
||
"]\n",
|
||
"\n",
|
||
"print(\"📊 RAG Evaluation Results\\n\" + \"=\" * 60)\n",
|
||
"\n",
|
||
"for case in test_cases:\n",
|
||
" question = case[\"question\"]\n",
|
||
"\n",
|
||
" # Retrieve and validate question\n",
|
||
" question_raw = case.get(\"question\", \"\")\n",
|
||
" if not isinstance(question_raw, str):\n",
|
||
" raise TypeError(\"case['question'] must be a string\")\n",
|
||
" question = question_raw\n",
|
||
"\n",
|
||
" # Retrieve context\n",
|
||
" docs = vector_store.similarity_search(question, k=3)\n",
|
||
" context = format_docs(docs)\n",
|
||
"\n",
|
||
" # Get or fake the answer\n",
|
||
" use_rag = case.get(\"use_rag\", False) is True\n",
|
||
" fake_answer = case.get(\"fake_answer\", \"\")\n",
|
||
" fake_answer_str = fake_answer if isinstance(fake_answer, str) else str(fake_answer)\n",
|
||
" answer = rag_assistant(question, k=3) if use_rag else fake_answer_str\n",
|
||
"\n",
|
||
" # Evaluate\n",
|
||
" try:\n",
|
||
" scores = evaluate_rag_answer(question, context, answer)\n",
|
||
" except (json.JSONDecodeError, TypeError, ValueError):\n",
|
||
" raw_eval = llm.invoke(\n",
|
||
" f\"\"\"Evaluate this RAG answer and return ONLY valid JSON.\n",
|
||
"\n",
|
||
"QUESTION: {question}\n",
|
||
"\n",
|
||
"RETRIEVED CONTEXT:\n",
|
||
"{context}\n",
|
||
"\n",
|
||
"GENERATED ANSWER:\n",
|
||
"{answer}\n",
|
||
"\n",
|
||
"Return exactly:\n",
|
||
"{{\"faithfulness\": <int 0-5>, \"relevance\": <int 0-5>, \"feedback\": \"<short explanation>\"}}\"\"\",\n",
|
||
" )\n",
|
||
" raw_text = raw_eval.content if isinstance(raw_eval.content, str) else str(raw_eval.content)\n",
|
||
" start = raw_text.find(\"{\")\n",
|
||
" end = raw_text.rfind(\"}\") + 1\n",
|
||
"\n",
|
||
" if start != -1 and end > start:\n",
|
||
" scores = json.loads(raw_text[start:end])\n",
|
||
" else:\n",
|
||
" scores = {\n",
|
||
" \"faithfulness\": 0,\n",
|
||
" \"relevance\": 0,\n",
|
||
" \"feedback\": \"Could not parse evaluator response as JSON.\",\n",
|
||
" }\n",
|
||
"\n",
|
||
" print(f'\\n{case['label']}: \"{question}\"')\n",
|
||
" print(f\" Faithfulness : {scores.get('faithfulness', '?')}/5\")\n",
|
||
" print(f\" Relevance : {scores.get('relevance', '?')}/5\")\n",
|
||
" print(f\" Feedback : {scores.get('feedback', '?')}\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"\n",
|
||
"## 💭 Final Thoughts\n",
|
||
"\n",
|
||
"<div style=\"background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); color: white; padding: 25px; border-radius: 15px; margin: 20px 0;\">\n",
|
||
" <h3 style=\"margin-top: 0;\">🎓 Remember:</h3>\n",
|
||
" <p><strong>\"The best RAG system is the one that solves your specific problem.\"</strong></p>\n",
|
||
" <ul>\n",
|
||
" <li>Start simple and iterate</li>\n",
|
||
" <li>Measure what matters to your users</li>\n",
|
||
" <li>Focus on retrieval quality first</li>\n",
|
||
" <li>Then optimize generation</li>\n",
|
||
" <li>Always test with real users</li>\n",
|
||
" </ul>\n",
|
||
"</div>\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"<div style=\"text-align: center; background: linear-gradient(135deg, #FF6B6B 0%, #FFE66D 100%); padding: 30px; border-radius: 15px; margin: 30px 0;\">\n",
|
||
" <h2 style=\"margin: 0; font-size: 2.5em;\">🏅 You're Now (Nearly) aRAG Expert!</h2>\n",
|
||
" <p style=\"margin: 20px 0 0 0; font-size: 1.3em; font-weight: bold;\">Go forth and build amazing AI-powered systems! 🚀</p>\n",
|
||
"</div>\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"### 📚 Additional Resources:\n",
|
||
"\n",
|
||
"- [LangChain Documentation](https://python.langchain.com/)\n",
|
||
"- [RAG Best Practices](https://www.anthropic.com/research)\n",
|
||
"\n",
|
||
"### 🤝 Community:\n",
|
||
"\n",
|
||
"- LangChain Discord\n",
|
||
"- r/MachineLearning\n",
|
||
"- AI Stack Exchange\n",
|
||
"- Hugging Face Forums\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**Happy Building! 🎉**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": []
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "studies (3.13.9)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.13.9"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 4
|
||
}
|