# In-line Evaluations

In-line evaluations allow you to assess and score agent behavior as it happens—directly within the execution flow of your agent. Unlike post-hoc evaluations, which run after an agent completes its task, in-line evaluations provide real-time feedback by observing inputs, intermediate steps, or outputs during execution.

These evaluations can:

- Score individual steps such as retrieval or generation
- Detect recall issues, hallucinations or safety issues
- Affect agent orchestration by modifying the agent's state

By integrating evaluations into the runtime loop, agents can become more self-aware, adaptive, and robust, especially in complex or dynamic tasks.

`TruLens` inline evaluations perform two critical steps:

1. Execute an evaluation
2. Add the evaluation results to the agent's state

Consider a `LangGraph` agent with the following instrumented research node.

Example

```
@instrument(
    span_type=SpanAttributes.SpanType.RETRIEVAL,
    attributes=lambda ret, exception, *args, **kwargs: {
        SpanAttributes.RETRIEVAL.QUERY_TEXT: args[0]["messages"][-1].content,
        SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: [\
            json.loads(dumps(message)).get("kwargs", {}).get("content", "")\
            for message in ret.update["messages"]\
            if isinstance(message, ToolMessage)\
        ]
        if hasattr(ret, "update")
        else "No tool call",
    },
)
def research_node(
    state: MessagesState,
) -> Command[Literal["chart_generator", END]]:
    result = research_agent.invoke(state)
    goto = get_next_node(result["messages"][-1], "chart_generator")
    # wrap in a human message, as not all providers allow
    # AI message at the last position of the input messages list
    result["messages"][-1] = HumanMessage(
        content=result["messages"][-1].content, name="researcher"
    )
    return Command(
        update={
            # share internal message history of research agent with other agents
            "messages": result["messages"],
        },
        goto=goto,
    )
```

In this example, we can define a feedback function that accepts the `research_node`'s instrumented span attributes: `QUERY_TEXT` and `RETRIEVED_CONTEXTS`.

Example

```
f_context_relevance = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Inline Context Relevance",
    selectors={
        "question": Selector(
            span_type=SpanAttributes.SpanType.RETRIEVAL,
            span_attribute=SpanAttributes.RETRIEVAL.QUERY_TEXT,
        ),
        "context": Selector(
            span_type=SpanAttributes.SpanType.RETRIEVAL,
            span_attribute=SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS,
            collect_list=False,
        ),
    },
    agg=np.mean,
)
```

Then, once we have created a feedback function that operates on the instrumented span attributes for the method we want to evaluate, we can simply add the `@inline_evaluation` decorator with the feedback function we just created.

Example

```
@inline_evaluation(f_context_relevance)
@instrument(
    span_type=SpanAttributes.SpanType.RETRIEVAL,
    attributes=lambda ret, exception, *args, **kwargs: {
        SpanAttributes.RETRIEVAL.QUERY_TEXT: args[0]["messages"][-1].content,
        SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: [\
            json.loads(dumps(message)).get("kwargs", {}).get("content", "")\
            for message in ret.update["messages"]\
            if isinstance(message, ToolMessage)\
        ]
        if hasattr(ret, "update")
        else "No tool call",
    },
)
def research_node(
    state: MessagesState,
) -> Command[Literal["chart_generator", END]]:
    result = research_agent.invoke(state)
    goto = get_next_node(result["messages"][-1], "chart_generator")
    # wrap in a human message, as not all providers allow
    # AI message at the last position of the input messages list
    result["messages"][-1] = HumanMessage(
        content=result["messages"][-1].content, name="researcher"
    )
    return Command(
        update={
            # share internal message history of research agent with other agents
            "messages": result["messages"],
        },
        goto=goto,
    )
```

Note

Feedback functions used for inline evaluation must operate on available instrumented spans of the method that is being evaluated.

After the feedback function is executed, evaluation results will be added to the state. Inline evaluations are currently only implemented for `LangGraph`, additional framework support will follow.

LangGraph-specific Implementation Details

In `LangGraph`, the evaluations are formatted as `AnyMessage` objects and appended to the `messages` key in `MessageState.

By adding the evaluation results to the agent's state, the agent can then use evaluation results to guide execution steps. For example, by informing the agent that an initial retrieval step lacks context relevance, the agent may choose to perform additional research before moving on to generate a final answer.