RAGAS
2. Install Dependencies and Import Libraries¶
Run the cell below to install Git LFS, which we use to download our dataset.
!git lfs install
Install and import Python dependencies.
!pip install "ragas==0.1.4" pypdf "arize-phoenix[llama-index]" "openai>=1.0.0" pandas
import pandas as pd
# Display the complete contents of dataframe cells.
pd.set_option("display.max_colwidth", None)
3. Setup¶
Set your OpenAI API key if it is not already set as an environment variable.
import os
from getpass import getpass
if not (openai_api_key := os.getenv("OPENAI_API_KEY")):
openai_api_key = getpass("🔑 Enter your OpenAI API key: ")
os.environ["OPENAI_API_KEY"] = openai_api_key
Launch Phoenix in the background and setup auto-instrumentation for llama-index and LangChain so that your OpenInference spans and traces are sent to and collected by Phoenix. OpenInference is an open standard built atop OpenTelemetry that captures and stores LLM application executions. It is designed to be a category of telemetry data that is used to understand the execution of LLMs and the surrounding application context, such as retrieval from vector stores and the usage of external tools such as search engines or APIs.
import phoenix as px
(session := px.launch_app()).view()
from openinference.instrumentation.langchain import LangChainInstrumentor
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import SpanLimits, TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
endpoint = "http://127.0.0.1:6006/v1/traces"
tracer_provider = TracerProvider(span_limits=SpanLimits(max_attributes=100_000))
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint)))
LlamaIndexInstrumentor().instrument(skip_dep_check=True, tracer_provider=tracer_provider)
LangChainInstrumentor().instrument(skip_dep_check=True, tracer_provider=tracer_provider)
4. Generate Your Synthetic Test Dataset¶
Curating a golden test dataset for evaluation can be a long, tedious, and expensive process that is not pragmatic — especially when starting out or when data sources keep changing. This can be solved by synthetically generating high quality data points, which then can be verified by developers. This can reduce the time and effort in curating test data by 90%.
Run the cell below to download a dataset of prompt engineering papers in PDF format from arXiv and read these documents using LlamaIndex.
!git clone https://huggingface.co/datasets/explodinggradients/prompt-engineering-papers
from llama_index.core import SimpleDirectoryReader
dir_path = "./prompt-engineering-papers"
reader = SimpleDirectoryReader(dir_path, num_files_limit=2)
documents = reader.load_data()
An ideal test dataset should contain data points of high quality and diverse nature from a similar distribution to the one observed during production. Ragas uses a unique evolution-based synthetic data generation paradigm to generate questions that are of the highest quality which also ensures diversity of questions generated. Ragas by default uses OpenAI models under the hood, but you’re free to use any model of your choice. Let’s generate 100 data points using Ragas.
from phoenix.trace import using_project
from ragas.testset.evolutions import multi_context, reasoning, simple
from ragas.testset.generator import TestsetGenerator
TEST_SIZE = 5
# generator with openai models
generator = TestsetGenerator.with_openai()
# set question type distribution
distribution = {simple: 0.5, reasoning: 0.25, multi_context: 0.25}
# generate testset
with using_project("ragas-testset"):
testset = generator.generate_with_llamaindex_docs(
documents, test_size=TEST_SIZE, distributions=distribution
)
test_df = (
testset.to_pandas().sort_values("question").drop_duplicates(subset=["question"], keep="first")
)
test_df.head(2)
You are free to change the question type distribution according to your needs. Since we now have our test dataset ready, let’s move on and build a simple RAG pipeline using LlamaIndex.
5. Build Your RAG Application With LlamaIndex¶
LlamaIndex is an easy to use and flexible framework for building RAG applications. For the sake of simplicity, we use the default LLM (gpt-3.5-turbo) and embedding models (openai-ada-2).
from llama_index.core import VectorStoreIndex
from llama_index.embeddings.openai import OpenAIEmbedding
from phoenix.trace import using_project
def build_query_engine(documents):
vector_index = VectorStoreIndex.from_documents(
documents,
embed_model=OpenAIEmbedding(),
)
query_engine = vector_index.as_query_engine(similarity_top_k=2)
return query_engine
with using_project("indexing"):
# By assigning a project name, the instrumentation will send all the embeddings to the indexing project
query_engine = build_query_engine(documents)
If you check Phoenix, you should see embedding spans from when your corpus data was indexed. Export and save those embeddings into a dataframe for visualization later in the notebook.
from time import sleep
# Wait a little bit in case data hasn't beomme fully available
sleep(2)
from phoenix.trace.dsl.helpers import SpanQuery
client = px.Client()
corpus_df = px.Client().query_spans(
SpanQuery().explode(
"embedding.embeddings",
text="embedding.text",
vector="embedding.vector",
),
project_name="indexing",
)
corpus_df.head(2)
6. Evaluate Your LLM Application¶
Ragas provides a comprehensive list of metrics that can be used to evaluate RAG pipelines both component-wise and end-to-end.
To use Ragas, we first form an evaluation dataset comprised of a question, generated answer, retrieved context, and ground-truth answer (the actual expected answer for the given question).
import pandas as pd
from datasets import Dataset
from phoenix.trace import using_project
from tqdm.auto import tqdm
def generate_response(query_engine, question):
response = query_engine.query(question)
return {
"answer": response.response,
"contexts": [c.node.get_content() for c in response.source_nodes],
}
def generate_ragas_dataset(query_engine, test_df):
test_questions = test_df["question"].values
responses = [generate_response(query_engine, q) for q in tqdm(test_questions)]
dataset_dict = {
"question": test_questions,
"answer": [response["answer"] for response in responses],
"contexts": [response["contexts"] for response in responses],
"ground_truth": test_df["ground_truth"].values.tolist(),
}
ds = Dataset.from_dict(dataset_dict)
return ds
with using_project("llama-index"):
ragas_eval_dataset = generate_ragas_dataset(query_engine, test_df)
ragas_evals_df = pd.DataFrame(ragas_eval_dataset)
ragas_evals_df.head(2)
Check out Phoenix to view your LlamaIndex application traces.
print(session.url)
We save out a couple of dataframes, one containing embedding data that we'll visualize later, and another containing our exported traces and spans that we plan to evaluate using Ragas.
from time import sleep
# Wait a bit in case data hasn't beomme fully available
sleep(2)
# dataset containing embeddings for visualization
query_embeddings_df = px.Client().query_spans(
SpanQuery().explode("embedding.embeddings", text="embedding.text", vector="embedding.vector"),
project_name="llama-index",
)
query_embeddings_df.head(2)
from phoenix.session.evaluation import get_qa_with_reference
# dataset containing span data for evaluation with Ragas
spans_dataframe = get_qa_with_reference(client, project_name="llama-index")
spans_dataframe.head(2)
Ragas uses LangChain to evaluate your LLM application data. Since we initialized the LangChain instrumentation above we can see what's going on under the hood when we evaluate our LLM application.
Evaluate your LLM traces and view the evaluation scores in dataframe format.
from phoenix.trace import using_project
from ragas import evaluate
from ragas.metrics import (
answer_correctness,
context_precision,
context_recall,
faithfulness,
)
# Log the traces to the project "ragas-evals" just to view
# how Ragas works under the hood
with using_project("ragas-evals"):
evaluation_result = evaluate(
dataset=ragas_eval_dataset,
metrics=[faithfulness, answer_correctness, context_recall, context_precision],
)
eval_scores_df = pd.DataFrame(evaluation_result.scores)
Submit your evaluations to Phoenix so they are visible as annotations on your spans.
# Assign span ids to your ragas evaluation scores (needed so Phoenix knows where to attach the spans).
span_questions = (
spans_dataframe[["input"]]
.sort_values("input")
.drop_duplicates(subset=["input"], keep="first")
.reset_index()
.rename({"input": "question"}, axis=1)
)
ragas_evals_df = ragas_evals_df.merge(span_questions, on="question").set_index("context.span_id")
test_df = test_df.merge(span_questions, on="question").set_index("context.span_id")
eval_data_df = pd.DataFrame(evaluation_result.dataset)
eval_data_df = eval_data_df.merge(span_questions, on="question").set_index("context.span_id")
eval_scores_df.index = eval_data_df.index
query_embeddings_df = (
query_embeddings_df.sort_values("text")
.drop_duplicates(subset=["text"])
.rename({"text": "question"}, axis=1)
.merge(span_questions, on="question")
.set_index("context.span_id")
)
from phoenix.trace import SpanEvaluations
# Log the evaluations to Phoenix under the project "llama-index"
# This will allow you to visualize the scores alongside the spans in the UI
for eval_name in eval_scores_df.columns:
evals_df = eval_scores_df[[eval_name]].rename(columns={eval_name: "score"})
evals = SpanEvaluations(eval_name, evals_df)
px.Client().log_evaluations(evals)
If you check out Phoenix, you'll see your Ragas evaluations as annotations on your application spans.
print(session.url)
8. Recap¶
Congrats! You built and evaluated a LlamaIndex query engine using Ragas and Phoenix. Let's recap what we learned:
- With Ragas, you bootstraped a test dataset and computed metrics such as faithfulness and answer correctness to evaluate your LlamaIndex query engine.
- With OpenInference, you instrumented your query engine so you could observe the inner workings of both LlamaIndex and Ragas.
- With Phoenix, you collected your spans and traces, imported your evaluations for easy inspection, and visualized your embedded queries and retrieved documents to identify pockets of poor performance.
This notebook is just an introduction to the capabilities of Ragas and Phoenix. To learn more, see the Ragas and Phoenix docs.
If you enjoyed this tutorial, please leave a ⭐ on GitHub: