This guide explains how to configure A/B testing of conversational agents using
CCAI Platform.

## Why A/B test virtual agents?

A/B testing (or split testing) is a powerful methodology for optimizing the
performance of conversational AI. Instead of relying on intuition when updating
a flow, you can let real-time data guide your decisions.

Core benefits include the following:

- **Risk mitigation:** Test a new "Agent B" with a small percentage of traffic (for example, 10%) before a full rollout to ensure that it doesn't negatively impact containment rates.
- **Performance comparison:** Compare specific KPIs such as Intent Matching Accuracy, Task Completion Rate, and Customer Sentiment across different versions.
- **Iterative refinement:** Experiment with different prompts, personas, or LLM-backed generators to find the most effective way to resolve user queries.

## Configure testing

This section explains how to configure CCAI Platform to support A/B
testing for the capabilities of an agent. This applies to both voice agents and
digital channels such as mobile and web.

### Prepare your virtual agents

Before configuring the telephony or routing, ensure you have two separate
virtual agent targets ready in your environment. Refer to the documentation for
configuring virtual agents. You will likely need two
separate conversational profiles. The actual conversational agents could be
separate agents or they could be the same agent with a different environments.

- **Agent A (control):** Your existing, stable virtual agent.

- **Agent B (variant):** The agent containing the new features, prompts, or
  logic changes that you want to test.

### Create dedicated CCAI Platform queues

In the CCAI Platform portal, you need to establish a hierarchy of
queues to manage the traffic flow. This will depend on whether you are using voice or
digital channels. Refer to the documentation for further details on creating
queues.

1. **Create queue A:** Navigate to Queue Management. Create a new queue named
   Queue_A_Test. Assign virtual agent A as the primary handler for this queue.

2. **Create queue B:** Create a second queue named Queue_B_Test. Assign virtual agent B
   as the primary handler for this queue.

3. **Create parent (entry) queue:** Create a queue named Main_Entry_Point. This
   queue won't have a virtual agent assigned directly; it will act as the
   "traffic splitter."

## Implement routing logic

To distribute traffic, you must configure the routing rules in the
Main_Entry_Point queue. This will leverage the automatic redirection feature to
specify the routing groups. Each routing group will include the routing to the
relative queue. The percentage allocation will also be associated with the
specific routing group. Refer to the documentation for further details on
configuring the routing rules.

### Define the traffic split

You will use a percentage-based distribution rule. This lets you control exactly
how much volume each agent receives.

| **Component** | Logic | Goal |
|---|---|---|
| **Trigger** | All incoming calls or chats to `Main_Entry_Point`. | Catch all traffic. |
| **Action 1** | Route **X** % of traffic to `Queue_A_Test`. | Establish the control group. |
| **Action 2** | Route Remaining % of traffic to `Queue_B_Test`. | Establish the test group. |

### Configuration workflow

1. Open the **Automatic Redirection** configuration for your `Main_Entry_Point` queue.
2. Add a **group**.
3. Select the **redirection selection** of Queue.
4. Select the queue that should be associated with this routing group A.
5. Select **Percentage Split**.
6. Select **Add**.
7. Add a **group**.
8. Select the **redirection selection** of Queue.
9. Select the queue that should be associated with this routing group B.
10. Select **Percentage Split**.
11. Select **Add**.
12. Define your ratio (for example, 50/50 for a balanced test, or 90/10 for a canary release).
13. Save the configuration.

## Monitor and analyze

When the traffic is flowing, use the CCaaS Analytics Dashboard to compare the
performance.

- Filter reports by `Queue_A_Test` versus `Queue_B_Test`.
- Look for statistical significance in **Average Handle Time (AHT)** and **Transfer Rate to Human Agent**.
- Look at the conversational agent analytics to look for any additional details that would indicate a positive or negative impact to performance.