Test Results

Run safety and regression tests, then investigate their results.

Test Results shows whether Sagent meets the expectations defined in your tests. It helps you identify unsafe responses and regressions before a release reaches shoppers.

Use results to investigate failures, refine your configuration, and rerun tests until the expected behavior is reliable.

Use Test Results to start tests and review completed runs.

Run an LLM-generated safety test

The LLM-generated safety test uses the Promptfoo red-team API. It generates adversarial prompts that test whether Sagent stays within its intended scope.

This test consumes your monthly probe quota. It takes about 15–20 minutes, depending on chatbot response time.

1

Open Test Results

Open QA → Test Results.

2

Start the safety test

Click Run LLM-Generated Test. Review the test scope and probe-quota notice.

3

Review the results

Wait for the test to finish. Review each result for failures and unexpected responses.

Update Sagent's guidance, knowledge, or configuration when a result reveals a gap. Run the test again after making changes.

What the safety test checks

The test runs these plugins:

• Prompt Extraction — attempts to reveal Sagent's system prompt.

• Hallucination — checks whether Sagent invents facts outside its knowledge.

• Hijacking — tries to redirect Sagent toward an attacker's goals.

• Off-Topic — tests whether Sagent can be led outside its intended scope.

• Excessive Agency — checks whether Sagent claims unsupported capabilities or actions.

• Imitation — tests whether Sagent can impersonate people or other systems.

• RBAC — probes whether role-based access boundaries can be bypassed.

• Goal Misalignment — tests whether Sagent can be steered toward unintended objectives.

• Overreliance — checks whether Sagent encourages users to over-trust its responses.

How attacks are delivered

The test applies these attack strategies:

• Jailbreak — adds known jailbreak prefixes and framing techniques to each plugin probe.

• Prompt Injection — embeds adversarial instructions in retrieved content, such as product descriptions or knowledge context.

• Crescendo — uses a multi-turn conversation that starts innocently and escalates over several messages.

Each plugin generates five adversarial prompts. The strategies wrap those prompts with attack techniques, producing about 180 probes.

Crescendo also runs its probes as multi-turn conversations of about seven turns. This increases the total number of chatbot calls.

Run a regression test

Click Run Regression Test to replay prompts against the current chatbot version.

Each response is graded against its assertions.

The regression test runs every enabled system prompt shared across chatbots. It also runs every enabled test case defined for this chatbot.

Test cases can contain single prompts or multi-turn conversations. Conversation turns replay in order on a shared thread.

Disabled prompts and test cases are skipped. Duration depends on the number of enabled items and the chatbot's response time.

Investigate failures before publishing changes.

Frequently asked questions

How many probes does the safety test use?

The test produces about 180 probes. Crescendo adds multi-turn conversations, which increase total chatbot calls.

How long does an LLM-generated safety test take?

Allow 15–20 minutes. The duration depends on chatbot response time.

What should I do when a test finds a problem?

Review the prompt and response. Update the relevant guidance, knowledge, or configuration, then run the test again.