Generating Test Cases

Generating a sufficient test set for evaluating an app is an early step in the development cycle.

TruLens allows you to generate a test set of a specified breadth and depth, tailored to your app and data. The resulting test set will be a list of test prompts of length depth, for breadth categories of prompts. This test set will be made up of breadth X depth prompts organized by prompt category.

Initial

pip install "trulens-benchmark"
pip install trulens-providers-openai

Note: Since we will using a LLM provider to generate the access, pick your provider and setup the environment variable, say OpenAI.

Example

import os
from trulens.benchmark.generate.generate_test_set import GenerateTestSet
from trulens.providers.openai import OpenAI

os.environ["OPENAI_API_KEY"] = "<replace-with-your-api-key>"

# Considering you have your LangChain rag_chain in place.

test = GenerateTestSet(app_callable = rag_chain.invoke)
test_set = test.generate_test_set(
  test_breadth = 3,
  test_depth = 2
)
test_set

Returns:

{'Code implementation': [\
  'What are the steps to follow when implementing code based on the provided instructions?',\
  'What is the required format for each file when outputting the content, including all code?'\
  ],
 'Short term memory limitations': [\
  'What is the capacity of short-term memory and how long does it last?',\
  'What are the two subtypes of long-term memory and what types of information do they store?'\
  ],
 'Planning and task decomposition challenges': [\
  'What are the challenges faced by LLMs in adjusting plans when encountering unexpected errors during long-term planning?',\
  'How does Tree of Thoughts extend the Chain of Thought technique for task decomposition and what search processes can be used in this approach?'\
  ]
}

Optionally, you can also provide a list of examples (few-shot) to guide the LLM app to a particular type of question.

Example

examples = [\
  "What is sensory memory?",\
  "How much information can be stored in short term memory?"\
]

fewshot_test_set = test.generate_test_set(
  test_breadth = 3,
  test_depth = 2,
  examples = examples
)
fewshot_test_set

Returns:

{'Code implementation': [\
  'What are the subcategories of sensory memory?',\
  'What is the capacity of short-term memory according to Miller (1956)?'\
  ],
 'Short term memory limitations': [\
  'What is the duration of sensory memory?',\
  'What are the limitations of short-term memory in terms of context capacity?'\
  ],
 'Planning and task decomposition challenges': [\
  'How long does sensory memory typically last?',\
  'What are the challenges in long-term planning and task decomposition?'\
  ]
}

In combination with record metadata logging, this gives you the ability to understand the performance of your application across different prompt categories.

Example

with tru_recorder as recording:
    for category in test_set:
        recording.record_metadata=dict(prompt_category=category)
        test_prompts = test_set[category]
        for test_prompt in test_prompts:
            llm_response = rag_chain.invoke(test_prompt)