RAG Evaluation | Praveen's Portfolio

Why RAG Evaluation?

RAG needs eval, because it can generate shitty or "not make any sence" output if we don't eval the following three phases.

Ingestion
Retrieval
Generation

Why traditional RAG fails?

The retriever is used to get the relevant data to the user query from the vector store, this will fail to give the right data or sometimes zero relevant data, if there was no proper chunking and spliting of input data in the ingestion phase, to full fill the user query with the right answer the LLM starts to hallucinate and generate non sense outputs by pretending itself a subject expert.
Traditional RAG is enough if we are dealing with very small number of data.

In this blog we will learn how to eval the RAG system and yeild 10x better output than the traditional RAG system.

Let's start with the Ingestion Phase

Ingestion phase is very curcial since this is dealing with the embedding data and chunks.
In traditional chunking statergies we use to generate the chunks by number of token or by the end of a sentence, This can causes the high possible retriever failure for bringing the relevant data because these random spliting technique only split the data by hardcoded delimeter but not by the topic.
This gives the hight probability for retriving the most unrelevant data along with the relevant data as result just because both of the vectors is slightly similar with the query, This cause the token window blow away in a very short time.

enter image description here

How we going to solve this problem?

We can solve this problem by using the semantic chunking technique, in this technique we will split the data by the topic and not by the hardcoded delimeter, This will give us the most relevant data to the user query and also it will reduce the token window blow away.

Let's EVAL the ingestion phase so we can decide whether the chunking needs fix or not based on the EVAL score and outputs.

What are the parameters we going to eval in this phase?

export type SingleTurnResult = {
  recommendedThreshold: number
  boundaryScores: number[]
  withinTopicScores: number[]
  withinTopicMean: number
  boundaryMean: number
  gap: number
  midpoint: number
  confidence: 'high' | 'medium' | 'low'
  reason: string
}

We need to nomally chunk the data by either number of tokens or by the end of a sentence, and for those chunks we need to generate the vector embedding to find the similarity between the chunks
By using the cosineSimilarity function, We can store the similarity between the subsequent chunks from the chunks array in an resultant arry, This will give us the score of each pair of chunks and we can use these scores to determine the other parameters we seen before.
With this recommendedThreshold we can decide the threshold for the semantic chunking technique, Using this threshold we can do the chunk splitting by topic instead of using delimeter or max token limit.

here is the code that get us the above parameters:

export const singleTurnExecutor = async (
  text: string
): Promise<SingleTurnResult> => {
  // this generates chunk by the end of the sentence
  const chunks = generateChunks(text)

  const vectors = []

  for (let [i, chunk] of chunks.entries()) {
    vectors.push((await embed(chunk)).embeddings[0])
  }

  const scores = []

  for (let i = 0; i < vectors.length - 1; i++) {
    const score = cosineSimilarity(vectors[i] as any, vectors[i + 1] as any)
    scores.push(score)
  }

  scores.sort()

  const result = estimateResult(scores)

  return result
}

const estimateResult = (arr: number[]): SingleTurnResult => {
  // find the biggest gap
  let biggestGap = 0
  let gapIndex = 0

  for (let i = 0; i < arr.length - 1; i++) {
    const gap = (arr[i + 1] as number) - (arr[i] as number)
    if (gap > biggestGap) {
      biggestGap = gap
      gapIndex = i
    }
  }

  //everything below the gap is boundaryScores
  const boundaryScores = arr.slice(0, gapIndex + 1)

  //everything above the gap is withinTopicScores
  const withinTopicScores = arr.slice(gapIndex + 1)

  //boundaryMean
  const boundaryMean =
    boundaryScores.reduce((a, b) => a + b, 0) / boundaryScores.length
  //withinTopicMean
  const withinTopicMean =
    withinTopicScores.reduce((a, b) => a + b, 0) / withinTopicScores.length

  const lowestWithinTopic = Math.min(...withinTopicScores)
  const highestBoundary = Math.max(...boundaryScores)

  console.log(lowestWithinTopic, highestBoundary)

  const gap = lowestWithinTopic - highestBoundary

  //midpoint
  const midpoint = (lowestWithinTopic + highestBoundary) / 2

  // bias upward by 10% of the gap to avoid false splits
  const recommendedThreshold = Math.round((midpoint + gap * 0.1) * 100) / 100

  let confidence: 'high' | 'medium' | 'low'
  let reason: string

  if (gap >= 0.25) {
    confidence = 'high'
    reason = `Clear separation of ${gap.toFixed(2)} between clusters. 
              Threshold is reliable.`
  } else if (gap >= 0.1) {
    confidence = 'medium'
    reason = `Moderate separation of ${gap.toFixed(2)}. 
              Consider running on more documents to confirm.`
  } else {
    confidence = 'low'
    reason = `Weak separation of ${gap.toFixed(2)}. 
              Topics may be too similar or chunking is already good. 
              Try a different embedding model.`
  }

  return {
    recommendedThreshold,
    boundaryScores,
    withinTopicScores,
    confidence,
    boundaryMean,
    withinTopicMean,
    gap,
    midpoint,
    reason,
  }
}

EVAL Dashboard

For the EVAL Dashboard we can use the Lamniar dashboard to visualize the results of the EVAL and also to compare the results of the traditional RAG and the semantic chunking RAG.

Here is the result we got from the EVAL dashboard for the traditional RAG for the ingestion phase:

enter image description here

The above images shows the result of the EVAL using the test data where the information is not well structured and the topics are mixed together, as we can see the recommended threshold is very low and the confidence is low.

Threshold - 0.32, this means that the chunks are not well separated by the topic and there is a high probability for the retriever to bring the unrelevant data along with the relevant data, this will cause the LLM to generate bad response.

enter image description here

The above image shows the result of the EVAL using the test data where the information is well structured and the gap between the topics are very low, as we can see the recommended threshold is high and the confidence is high.

Threshold - 0.1, this means that the chunks are well separated by the topic and there is a low probability for the retriever to bring the unrelevant data along with the relevant data, this will cause the LLM to generate good response.

Why RAG Evaluation?

Why traditional RAG fails?

Let's start with the Ingestion Phase

Let's EVAL the ingestion phase so we can decide whether the chunking needs fix or not based on the EVAL score and outputs.

EVAL Dashboard

Here is the result we got from the EVAL dashboard for the traditional RAG for the ingestion phase:

Retrieval Phase...