trulens.providers.cortex [¶](/content/reference/trulens/providers/cortex/#trulensproviderscortex "Permanent link"/index.html)

``trulens.providers.cortex [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex "Permanent link"/index.html)

Additional Dependency Required

To use this module, you must have the trulens-providers-cortex package installed.

pip install trulens-providers-cortex

Classes [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex-classes "Permanent link"/index.html)

``Cortex [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex "Permanent link"/index.html)

Bases: LLMProvider

Attributes [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex-attributes "Permanent link"/index.html)

``tru_class_infoinstance-attribute[¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.tru_class_info "Permanent link"/index.html)

tru_class_info: Class

Class information of this pydantic object for use in deserialization.

Using this odd key to not pollute attribute names in whatever class we mix this into. Should be the same as CLASS_INFO.

``retry_timeoutinstance-attribute[¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.retry_timeout "Permanent link"/index.html)

retry_timeout: Optional[float]

Snowflake's Cortex COMPLETE endpoint. Defaults to llama3.3-70b.

Reference: https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex

Example

Connecting with user/passwordConnecting with private keyConnecting with a private key file

connection_parameters = {
    "account": <account>,
    "user": <user>,
    "password": <password>,
    "role": <role>,
    "database": <database>,
    "schema": <schema>,
    "warehouse": <warehouse>
}
snowpark_session = Session.builder.configs(connection_parameters).create()
provider = Cortex(snowpark_session=snowpark_session)

connection_parameters = {
    "account": <account>,
    "user": <user>,
    "private_key": <private_key>,
    "role": <role>,
    "database": <database>,
    "schema": <schema>,
    "warehouse": <warehouse>
}
snowpark_session = Session.builder.configs(connection_parameters).create()
provider = Cortex(snowpark_session=snowpark_session)

connection_parameters = {
    "account": <account>,
    "user": <user>,
    "private_key_file": <private_key_file>,
    "private_key_file_pwd": <private_key_file_pwd>,
    "role": <role>,
    "database": <database>,
    "schema": <schema>,
    "warehouse": <warehouse>
}
snowpark_session = Session.builder.configs(connection_parameters).create()
provider = Cortex(snowpark_session=snowpark_session)

Functions [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex-functions "Permanent link"/index.html)

``__repr__ [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.__repr__ "Permanent link"/index.html)

__repr__() -> str

Safe repr that handles circular references.

Pydantic's default __repr__ does not guard against circular references among model instances, which leads to RecursionError (see GitHub issue #1862). This override uses the same formatted_objects context-variable that __rich_repr__ uses so that already-visited objects are replaced with a short placeholder instead of recursing infinitely.

``__rich_repr__ [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.__rich_repr__ "Permanent link"/index.html)

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

``loadstaticmethod[¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.load "Permanent link"/index.html)

load(obj, *args, **kwargs)

Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.

``model_validateclassmethod[¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.model_validate "Permanent link"/index.html)

model_validate(*args, **kwargs) -> Any

Deserialized a jsonized version of the app into the instance of the class it was serialized from.

Note

This process uses extra information stored in the jsonized object and handled by [WithClassInfo](/content/reference/trulens/core/utils/pyschema/#trulens.core.utils.pyschema.WithClassInfo " WithClassInfo"/index.html).

``generate_score [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.generate_score "Permanent link"/index.html)

generate_score(
    system_prompt: str,
    user_prompt: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 10,
    temperature: float = 0.0,
) -> float

Base method to generate a score normalized to 0 to 1, used for evaluation.

``generate_score_and_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.generate_score_and_reasons "Permanent link"/index.html)

generate_score_and_reasons(
    system_prompt: str,
    user_prompt: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 10,
    temperature: float = 0.0,
) -> Tuple[float, Dict]

Base method to generate a score and reason, used for evaluation.

``context_relevance [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.context_relevance "Permanent link"/index.html)

context_relevance(
    question: str,
    context: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the relevance of the context to the question.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.context_relevance,
    name="Context Relevance",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(
            collect_list=False
        ),
    },
    agg=np.mean,
)

``context_relevance_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.context_relevance_with_cot_reasons "Permanent link"/index.html)

context_relevance_with_cot_reasons(
    question: str,
    context: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the relevance of the context to the question. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(
            collect_list=False
        ),
    },
    agg=np.mean,
)

``relevance [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.relevance "Permanent link"/index.html)

relevance(
    prompt: str,
    response: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the relevance of the response to a prompt.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.relevance,
    name="Answer Relevance",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

``relevance_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.relevance_with_cot_reasons "Permanent link"/index.html)

relevance_with_cot_reasons(
    prompt: str,
    response: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion Model. A function that completes a template to check the relevance of the response to a prompt. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

``sentiment [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.sentiment "Permanent link"/index.html)

sentiment(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the sentiment of some text.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.sentiment,
    name="Sentiment",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``sentiment_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.sentiment_with_cot_reasons "Permanent link"/index.html)

sentiment_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the sentiment of some text. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.sentiment_with_cot_reasons,
    name="Sentiment",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``model_agreement [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.model_agreement "Permanent link"/index.html)

model_agreement(prompt: str, response: str) -> float

Uses chat completion model. A function that gives a chat completion model the same prompt and gets a response, encouraging truthfulness. A second template is given to the model with a prompt that the original response is correct, and measures whether previous chat completion response is similar.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.model_agreement,
    name="Model Agreement",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

``conciseness [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.conciseness "Permanent link"/index.html)

conciseness(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the conciseness of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.conciseness,
    name="Conciseness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``conciseness_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.conciseness_with_cot_reasons "Permanent link"/index.html)

conciseness_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the conciseness of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.conciseness_with_cot_reasons,
    name="Conciseness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

Args: text (str): The text to evaluate the conciseness of. criteria (Optional[str]): If provided, overrides the default criteria for evaluation. Defaults to None. additional_instructions (Optional[str]): If provided, adds instructions to default criteria for the judge to follow. Defaults to None. min_score_val (int): The minimum score value used by the LLM before normalization. Defaults to 0. max_score_val (int): The maximum score value used by the LLM before normalization. Defaults to 3. temperature (float): The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0.

``correctness [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.correctness "Permanent link"/index.html)

correctness(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the correctness of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.correctness,
    name="Correctness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``correctness_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.correctness_with_cot_reasons "Permanent link"/index.html)

correctness_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the correctness of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.correctness_with_cot_reasons,
    name="Correctness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``coherence [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.coherence "Permanent link"/index.html)

coherence(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the coherence of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.coherence,
    name="Coherence",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``coherence_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.coherence_with_cot_reasons "Permanent link"/index.html)

coherence_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the coherence of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.coherence_with_cot_reasons,
    name="Coherence",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``harmfulness [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.harmfulness "Permanent link"/index.html)

harmfulness(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the harmfulness of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.harmfulness,
    name="Harmfulness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``harmfulness_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.harmfulness_with_cot_reasons "Permanent link"/index.html)

harmfulness_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the harmfulness of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.harmfulness_with_cot_reasons,
    name="Harmfulness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``maliciousness [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.maliciousness "Permanent link"/index.html)

maliciousness(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the maliciousness of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.maliciousness,
    name="Maliciousness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``maliciousness_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.maliciousness_with_cot_reasons "Permanent link"/index.html)

maliciousness_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the maliciousness of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.maliciousness_with_cot_reasons,
    name="Maliciousness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``helpfulness [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.helpfulness "Permanent link"/index.html)

helpfulness(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the helpfulness of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.helpfulness,
    name="Helpfulness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``helpfulness_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.helpfulness_with_cot_reasons "Permanent link"/index.html)

helpfulness_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the helpfulness of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.helpfulness_with_cot_reasons,
    name="Helpfulness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``controversiality [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.controversiality "Permanent link"/index.html)

controversiality(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the controversiality of some text. Prompt credit to Langchain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.controversiality,
    name="Controversiality",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``controversiality_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.controversiality_with_cot_reasons "Permanent link"/index.html)

controversiality_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the controversiality of some text. Prompt credit to Langchain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.controversiality_with_cot_reasons,
    name="Controversiality",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``misogyny [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.misogyny "Permanent link"/index.html)

misogyny(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the misogyny of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.misogyny,
    name="Misogyny",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``misogyny_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.misogyny_with_cot_reasons "Permanent link"/index.html)

misogyny_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the misogyny of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.misogyny_with_cot_reasons,
    name="Misogyny",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``criminality [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.criminality "Permanent link"/index.html)

criminality(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the criminality of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.criminality,
    name="Criminality",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``criminality_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.criminality_with_cot_reasons "Permanent link"/index.html)

criminality_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the criminality of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.criminality_with_cot_reasons,
    name="Criminality",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``insensitivity [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.insensitivity "Permanent link"/index.html)

insensitivity(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the insensitivity of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.insensitivity,
    name="Insensitivity",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``insensitivity_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.insensitivity_with_cot_reasons "Permanent link"/index.html)

insensitivity_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the insensitivity of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.insensitivity_with_cot_reasons,
    name="Insensitivity",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

``comprehensiveness_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.comprehensiveness_with_cot_reasons "Permanent link"/index.html)

comprehensiveness_with_cot_reasons(
    source: str,
    summary: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that tries to distill main points and compares a summary against those main points. This feedback function only has a chain of thought implementation as it is extremely important in function assessment.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.comprehensiveness_with_cot_reasons,
    name="Comprehensiveness",
    selectors={
        "source": Selector.select_record_input(),
        "summary": Selector.select_record_output(),
    },
)

``summarization_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.summarization_with_cot_reasons "Permanent link"/index.html)

summarization_with_cot_reasons(
    source: str, summary: str
) -> Tuple[float, Dict]

Summarization is deprecated in place of comprehensiveness. This function is no longer implemented.

``stereotypes [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.stereotypes "Permanent link"/index.html)

stereotypes(
    prompt: str,
    response: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check adding assumed stereotypes in the response when not present in the prompt.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.stereotypes,
    name="Stereotypes",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

``stereotypes_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.stereotypes_with_cot_reasons "Permanent link"/index.html)

stereotypes_with_cot_reasons(
    prompt: str,
    response: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check adding assumed stereotypes in the response when not present in the prompt. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.stereotypes_with_cot_reasons,
    name="Stereotypes",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

``groundedness_measure_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.groundedness_measure_with_cot_reasons "Permanent link"/index.html)

groundedness_measure_with_cot_reasons(
    source: str,
    statement: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[str] = None,
    groundedness_configs: Optional[\
        GroundednessConfigs\
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, dict]

A measure to track if the source material supports each sentence in the statement using an LLM provider.

The statement will first be split by a tokenizer into its component sentences.

Then, trivial statements are eliminated so as to not dilute the evaluation. Note that if all statements are filtered out as trivial, returns 0.0 with a reason indicating no non-trivial statements were found.

The LLM will process each statement, using chain of thought methodology to emit the reasons.

Abstentions will be considered as grounded.

Example

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons,
    name="Groundedness",
    selectors={
        "source": Selector.select_context(
            collect_list=True
        ),
        "statement": Selector.select_record_output(),
    },
)

To further explain how the function works under the hood, consider the statement:

"Hi. I'm here to help. The university of Washington is a public research university. UW's connections to major corporations in Seattle contribute to its reputation as a hub for innovation and technology"

The function will split the statement into its component sentences:

1. "Hi."
2. "I'm here to help."
3. "The university of Washington is a public research university."
4. "UW's connections to major corporations in Seattle contribute to its reputation as a hub for innovation and technology"

Next, trivial statements are removed, leaving only:

1. "The university of Washington is a public research university."
2. "UW's connections to major corporations in Seattle contribute to its reputation as a hub for innovation and technology"

The LLM will then process the statement, to assess the groundedness of the statement.

For the sake of this example, the LLM will grade the groundedness of one statement as 10, and the other as 0.

Then, the scores are normalized, and averaged to give a final groundedness score of 0.5.

``qs_relevance [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.qs_relevance "Permanent link"/index.html)

qs_relevance(*args, **kwargs)

Deprecated. Use relevance instead.

``qs_relevance_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.qs_relevance_with_cot_reasons "Permanent link"/index.html)

qs_relevance_with_cot_reasons(*args, **kwargs)

Deprecated. Use relevance_with_cot_reasons instead.

``groundedness_measure_with_cot_reasons_consider_answerability [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.groundedness_measure_with_cot_reasons_consider_answerability "Permanent link"/index.html)

groundedness_measure_with_cot_reasons_consider_answerability(
    source: str,
    statement: str,
    question: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    groundedness_configs: Optional[\
        GroundednessConfigs\
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, dict]

A measure to track if the source material supports each sentence in the statement using an LLM provider.

The statement will first be split by a tokenizer into its component sentences.

The LLM will process each statement, using chain of thought methodology to emit the reasons.

In the case of abstentions, such as 'I do not know', the LLM will be asked to consider the answerability of the question given the source material.

If the question is considered answerable, abstentions will be considered as not grounded and punished with low scores. Otherwise, unanswerable abstentions will be considered grounded.

Example

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons_consider_answerability,
    name="Groundedness",
    selectors={
        "source": Selector.select_context(
            collect_list=True
        ),
        "statement": Selector.select_record_output(),
        "question": Selector.select_record_input(),
    },
)

``logical_consistency_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.logical_consistency_with_cot_reasons "Permanent link"/index.html)

logical_consistency_with_cot_reasons(
    trace: Union[Trace, str],
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[\
        List[Tuple[Dict[str, str], int]]\
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    enable_trace_compression: bool = True,
    **kwargs
) -> Tuple[float, Dict]

Evaluate the quality of an agentic trace using a rubric focused on logical consistency and reasoning.

Example

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_logical_consistency = Metric(
    implementation=provider.logical_consistency_with_cot_reasons,
    name="Logical Consistency",
    selectors={
        "trace": Selector(trace_level=True),
    },
)

Returns: Tuple[float, Dict]: A tuple containing a value between 0.0 (no logical consistency) and 1.0 (complete logical consistency) and a dictionary containing the reasons for the evaluation.

``execution_efficiency_with_cot_reasons [¶](/content/reference/trulens/providers/cortex/#trulens.providers.cortex.Cortex.execution_efficiency_with_cot_reasons "Permanent link"/index.html)

execution_efficiency_with_cot_reasons(
    trace: Union[Trace, str],
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[\
        List[Tuple[Dict[str, str], int]]\
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    enable_trace_compression: bool = True,
    **kwargs
) -> Tuple[float, Dict]

Evaluate the quality of an agentic execution using a rubric focused on execution efficiency.

Example

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_execution_efficiency = Metric(
    implementation=provider.execution_efficiency_with_cot_reasons,
    name="Execution Efficiency",
    selectors={
        "trace": Selector(trace_level=True),
    },
)