📓 Custom Feedback Functions

Feedback functions are an extensible framework for evaluating LLMs.

The primary motivations for customizing feedback functions are either to improve alignment of an existing feedback function, or to evaluate on a new axis not addressed by an out-of-the-box feedback function.

Improving feedback function alignment through customization

Feedback functions can be customized through a number of parameter changes that influence score generation. For example, you can choose to run feedbacks with or without chain-of-thought reasoning, customize the output scale, or provide "few-shot" examples to guide alignment of a feedback function. All of these decisions affect the score generation and should be carefully tested and benchmarked.

Chain-of-thought Reasoning

Feedback functions can be run with chain-of-thought reasoning using their "with_cot_reasons" variant. Doing so provides both the benefit of a view into how the grading is performed, and improves alignment due to the auto-regressive nature of LLMs forcing the score to sequentially follow the reasons.

from trulens.core import Metric
from trulens.core import Selector
from trulens.providers.openai import OpenAI

provider = OpenAI(model_engine="gpt-4o")

provider.relevance(
    "What are the key considerations when starting a small business?",
    "Find a mentor who can guide you through the early stages and help you navigate common challenges.",
)

provider.relevance_with_cot_reasons(
    "What are the key considerations when starting a small business?",
    "Find a mentor who can guide you through the early stages and help you navigate common challenges.",
)

Output space

The output space is another very important variable to consider. This allows you to trade-off between a score's accuracy and granularity. The larger the output space, the lower the accuracy.

Output space can be modulated via the min_score_val and max_score_val keyword arguments.

The output space currently allows three selections:

• 0 or 1 (binary)
• 0 to 3 (default)
• 0 to 10

While the output you see is always on a scale from 0 to 1, changing the output space changes the score range prompting given to the LLM judge. The score produced by the judge is then scaled down appropriately.

For example, we can modulate the output space to 0-10.

provider.relevance(
    "What are the key considerations when starting a small business?",
    "Find a mentor who can guide you through the early stages and help you navigate common challenges.",
    min_score_val=0,
    max_score_val=10,
)

Temperature

When using LLMs, temperature is another parameter to be mindful of. Metrics default to a temperature of 0, but it can be useful in some cases to use higher temperatures, or even ensemble with metrics using different temperatures.

provider.relevance(
    "What are the key considerations when starting a small business?",
    "Find a mentor who can guide you through the early stages and help you navigate common challenges.",
    temperature=0.9,
)

Groundedness configurations

Groundedness has its own specific configurations that can be set with the GroundednessConfigs class.

from trulens.core.feedback import feedback

groundedness_configs = feedback.GroundednessConfigs(
    use_sent_tokenize=False, filter_trivial_statements=False
)

provider.groundedness_measure_with_cot_reasons(
    "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles.",
    "Hi, your football expert here. The first superbowl was held on Jan 15, 1967",
)

Custom Criteria

To customize the LLM-judge prompting, you can override standard criteria with your own custom criteria.

This can be useful to tailor LLM-judge prompting to your domain and improve alignment with human evaluations.

custom_relevance_criteria = """
A relevant response should provide a clear and concise answer to the question.
"""

provider.relevance(
    "What are the key considerations when starting a small business?",
    "Find a mentor who can guide you through the early stages and help you navigate common challenges.",
    criteria=custom_relevance_criteria,
    min_score_val=0,
    max_score_val=1,
)

Few-shot examples

You can also provide examples to customize metric scoring to your domain.

from trulens.feedback.v2 import feedback

fewshot_relevance_examples_list = [
    (
        {
            "query": "What are the key considerations when starting a small business?",
            "response": "You should focus on building relationships with mentors and industry leaders. Networking can provide insights, open doors to opportunities, and help you avoid common pitfalls.",
        },
        3,
    ),
]

provider.relevance(
    "What are the key considerations when starting a small business?",
    "Find a mentor who can guide you through the early stages and help you navigate common challenges.",
    examples=fewshot_relevance_examples_list,
)

Creating new custom metrics

You can add your own metrics to evaluate the qualities required by your application in two steps: by creating a new provider class and metric function in your notebook!

from trulens.core import Metric
from trulens.core import Provider
from trulens.core import Selector

class StandAlone(Provider):
    def custom_metric(self, my_text_field: str) -> float:
        return 1.0 / (1.0 + len(my_text_field) * len(my_text_field))

standalone = StandAlone()
f_custom_function = Metric(
    implementation=standalone.custom_metric,
    name="custom_feedback",
    selectors={
        "text": Selector.select_record_output(),
    },
)

Extending existing providers

In addition to calling your own methods, you can also extend stock feedback providers (such as OpenAI, AzureOpenAI, or Bedrock) to custom feedback implementations.

from trulens.providers.openai import AzureOpenAI

class CustomAzureOpenAI(AzureOpenAI):
    def style_check_professional(self, response: str) -> float:
        professional_prompt = str.format(
            "Please rate the professionalism of the following text on a scale from 0 to 10, where 0 is not at all professional and 10 is extremely professional: \n\n{}",
            response,
        )
        return self.generate_score(system_prompt=professional_prompt)

Learning to create and customize feedback functions not only enhances your evaluation process but ensures that you align more closely with specific needs and domains.