A/B Testing

Most AI builders pick their chatbot’s system prompt based on gut feeling. They write something that sounds good, deploy it, and hope for the best. But what if you could actually test two prompt variants on real users and measure which one performs better? This n8n workflow does exactly that: it randomly assigns users to either a baseline or alternative system prompt, remembers their assignment, and lets you collect data on which version gets better results. No guessing. Just data-driven prompt optimization.

Prefer to skip the setup? Grab the ready-made template and import it into your n8n instance in minutes — get the A/B testing template here.

What You’ll Build

By the end of this guide, you’ll have a fully functional A/B testing system that:

1. Accepts incoming chat messages with user session IDs
2. Stores two distinct system prompt variants (baseline and alternative)
3. Checks whether the session has been assigned to a test group before
4. Automatically assigns new users to one of the two variants using a 50/50 random split
5. Ensures returning users always see the same prompt variant they were originally assigned
6. Passes the correct prompt to your AI agent (OpenAI GPT-4o-mini)
7. Maintains full conversation history in PostgreSQL so the AI remembers previous messages
8. Records session and test assignment data for later analysis

How It Works — The Big Picture

Here’s the flow from incoming message to AI response:

┌────────────────────────────────────────────────────────────────┐
│  A/B TEST AI PROMPTS                                           │
│                                                                │
│  [Chat Trigger] → [Define Test Prompts] → [Check Session]     │
│                                              ↓                 │
│                                    [Session Assigned?]         │
│                                     ↙ Yes        ↘ No         │
│                              [Select Prompt]  [Assign Random]  │
│                                     ↘             ↙            │
│                                  [Select Prompt]               │
│                                       ↓                        │
│                                  [AI Agent]                    │
│                              (OpenAI + Memory)                 │
└────────────────────────────────────────────────────────────────┘

The workflow listens for incoming chat messages, queries Supabase to see if the user’s session already exists, and branches based on the result. New users get randomly assigned to one of two prompt variants; returning users get their original variant. Both paths converge at a single AI Agent node that uses the correct system prompt and maintains conversation memory through PostgreSQL.

What You’ll Need

• Supabase account (free tier is fine) — you’ll need a PostgreSQL database and the ability to run SQL queries
• OpenAI API key with access to GPT-4o-mini (cost: typically less than $1 per 1M tokens)
• n8n instance — either n8n Cloud (free or paid plans) or self-hosted
• Basic familiarity with n8n — understanding nodes, inputs, and outputs will help

Build time: 25-35 minutes from scratch, under 10 minutes if you import the ready-made template.

Step-by-Step Setup

1 Set Up Your Chat Trigger

Start with a Chat Trigger node (or HTTP Request if you’re building a custom endpoint). This node receives incoming user messages along with a session ID. The session ID is crucial — it’s how you identify repeat users.

Your incoming payload should look like this:

{
  "sessionId": "sess_7f3a2b91",
  "userId": "user_4c2e9k10",
  "message": "Hello, what's your recommendation for a CRM?"
}

2 Define Your Test Prompts

Add a Set node after the Chat Trigger. This node stores both system prompt variants. Here’s an example with a customer support chatbot:

Baseline prompt (friendly):

"You are a helpful customer support agent for an e-commerce platform. Be warm, approachable, and conversational. Always put the customer's needs first. If you don't know something, admit it and offer to escalate."

Alternative prompt (professional):

"You are a professional customer support specialist. Provide concise, accurate answers. Use technical terminology where appropriate. Focus on efficiency and quick resolution. Maintain professional boundaries while remaining courteous."

Store these as variables in your Set node. For example:

{
  "baseline_prompt": "You are a helpful customer support agent...",
  "alternative_prompt": "You are a professional customer support specialist..."
}

You can customize these prompts however you want — adjust tone, instructions, constraints, anything. The point is to test meaningful variations.

3 Query Supabase for Existing Sessions

Add a Supabase node (Query Rows) to check if this session has been assigned before. Set up the query like this:

Table: split_test_sessions

Filter: session_id = (incoming session_id)

This will return an empty array if the session is new, or one row if the session already exists. Save the result to a variable like session_lookup.

4 Add a Conditional: Is the Session Already Assigned?

Use an IF node to check whether the session exists:

session_lookup.length > 0

If true (session exists), branch to “Select Active Prompt”. If false (new session), branch to “Assign Random Variant”.

5 Assign a Random Variant to New Users

In the “false” branch, add a Function node that generates a random coin flip and inserts a new row into Supabase:

// Generate 50/50 random boolean
const show_alternative = Math.random() < 0.5;

// Return the assignment for the next node
return {
  show_alternative: show_alternative,
  session_id: $input.all()[0].json.sessionId,
  timestamp: new Date().toISOString()
};

Follow this with a Supabase Insert Rows node that saves the assignment to the database:

Table: split_test_sessions

Columns:

{
  "session_id": $input.all()[0].json.sessionId,
  "show_alternative": show_alternative,
  "created_at": new Date().toISOString()
}

6 Select the Active Prompt

Both paths (existing and new sessions) converge at a Set node that picks the correct system prompt. This node needs to check whether show_alternative is true or false and return the matching prompt:

{
  "system_prompt": $input.all()[0].json.show_alternative
    ? $input.all()[0].json.alternative_prompt
    : $input.all()[0].json.baseline_prompt
}

Make sure this node receives the show_alternative boolean from either the database query (existing session) or the assignment function (new session).

7 Configure the AI Agent with Memory

Add an OpenAI node configured as an AI Agent. Set it up like this:

Model: gpt-4o-mini

System prompt: Use the system_prompt variable from the previous Set node

Chat memory: Enable PostgreSQL memory using your Supabase connection. Configure it with:

• Connection: Your Supabase PostgreSQL connection
• Session ID: The incoming sessionId
• Memory type: Buffer memory or summarization (your choice based on conversation length)

This ensures the AI remembers all previous messages in the session, maintaining context across turns.

The Data Structure

You need a PostgreSQL table in Supabase to track session assignments. Here's the schema:

To create this table in Supabase, go to the SQL Editor and run:

CREATE TABLE split_test_sessions (
  id BIGSERIAL PRIMARY KEY,
  session_id TEXT NOT NULL UNIQUE,
  show_alternative BOOLEAN NOT NULL,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_session_id ON split_test_sessions(session_id);

The index on session_id speeds up lookups when users return.

Full System Flow with Data

Let's trace a complete example with realistic data:

1. User message arrives:

{
  "sessionId": "sess_7f3a2b91",
  "userId": "user_4c2e9k10",
  "message": "Hello, what CRM do you recommend for a 50-person startup?"
}

2. Query Supabase:

SELECT * FROM split_test_sessions WHERE session_id = 'sess_7f3a2b91';
// Result: [] (empty, this is a new user)

3. Assign random variant:

Math.random() < 0.5  // Returns true, assign alternative
// Insert into Supabase:
{
  "session_id": "sess_7f3a2b91",
  "show_alternative": true,
  "created_at": "2026-04-08T14:32:05.000Z"
}

4. Select the correct prompt:

// show_alternative is true, so use:
system_prompt = "You are a professional customer support specialist..."

5. AI Agent responds:

The OpenAI node receives the alternative system prompt and the user's message. With PostgreSQL memory enabled, it also pulls any previous messages from this session (none on first message). It generates a response:

"For a 50-person startup, I'd recommend HubSpot or Pipedrive. Both scale efficiently and offer the customization you'll need. What's your primary use case — sales pipeline or customer support?"

6. On the next turn (same session):

{
  "sessionId": "sess_7f3a2b91",
  "userId": "user_4c2e9k10",
  "message": "Mainly sales pipeline. What about pricing?"
}

Query returns the existing row with show_alternative: true. The same professional prompt is used. Memory context includes the entire conversation.

Testing Your Workflow

Before running it live, test each step:

1. Test the Chat Trigger: Send a sample message with a session ID through the webhook or chat interface. Check that the payload arrives correctly.
2. Test Supabase connectivity: Run a simple query (e.g., SELECT * FROM split_test_sessions LIMIT 1;) to verify the connection works.
3. Test the random assignment: Run the workflow 10 times with different session IDs and verify that Supabase records are created with roughly 50% true/false split.
4. Test the conditional logic: Create a session, run the workflow, then re-run with the same session ID. Verify that the second run retrieves the existing assignment instead of creating a new one.
5. Test the AI Agent: Verify the AI receives the correct system prompt by checking the API logs or n8n execution history.
6. Test memory persistence: Send multiple messages in the same session and confirm the AI remembers previous context.

Common Issues and Troubleshooting

Measuring Results

Now that your A/B test is running, how do you measure which prompt is better? A few approaches:

• User feedback: Add a thumbs-up/thumbs-down button after each AI response and record votes in a feedback table, tagged with session_id and show_alternative.
• Conversation length: Query Supabase to see average message count per session for each variant. Longer conversations might indicate more engaging prompts.
• Resolution time: If this is customer support, track how many turns it takes to resolve issues per variant.
• Manual review: Export a sample of responses from each variant and have a human evaluate quality, tone, and accuracy.
• Custom metrics: Log additional data (response time, token usage, user satisfaction score) to your Supabase table for analysis.

Run each variant for at least 100-200 sessions before drawing conclusions. Statistical significance matters.

Frequently Asked Questions

What's Next: Extending the Workflow

Once you have the basic A/B test working, consider these enhancements:

• Automatic winner selection: Set up a scheduled workflow that analyzes results every week and automatically switches all new users to the winning variant.
• Progressive rollout: Instead of 50/50, shift traffic gradually (90/10, 80/20, etc.) as one variant proves better.
• Segmented testing: Run different tests for different user segments (new vs. returning, by industry, by region).
• Prompt versioning: Store all prompt versions in Supabase with timestamps so you can track which exact variant each user saw.
• Multivariate testing: Test system prompt, temperature, and model all at once to find the optimal combination.
• Cost tracking: Log token usage per variant to see if one prompt is more efficient.