A/B Testing Emails: What to Test and How to Read the Results
On this page
A/B testing, also called split testing, means sending two versions of an email to different parts of your list and comparing how they perform. It replaces guesswork ("I think shorter subject lines work better") with evidence from your own audience. Most email tools include an A/B testing feature, so the hard part is not the technology. It is designing tests that give trustworthy answers, especially when your list is small.
How an A/B test works
- Pick one thing to change, such as the subject line.
- Create two versions (A and B) that are identical except for that one change.
- Split your audience randomly so each version goes to a similar group.
- Measure one main outcome, decided in advance.
- Compare the results and decide whether the difference is big enough to trust.
Many tools offer a "test then send" option: they send both versions to a small portion of your list, wait a set time, then send the apparent winner to everyone else. That is convenient, but with small test groups the "winner" may simply be luck. More on that below.
What to test
Focus on changes that could plausibly make a meaningful difference. Testing two nearly identical button colors rarely teaches you much on a small list.
| Element | Example test | Best measured by |
|---|---|---|
| Subject line | Question vs. statement; specific benefit vs. curiosity | Opens (with caution) or clicks |
| Sender name | Person's name vs. business name | Opens or clicks |
| Offer | Percentage discount vs. free shipping | Purchases or revenue |
| Call to action | "Book a call" vs. "See available times" | Clicks or bookings |
| Length and format | Short plain-text vs. longer designed email | Clicks or replies |
| Send time | Weekday morning vs. weekend | Clicks |
Choose the right metric
Decide before you send which number will decide the winner. Picking afterwards makes it too easy to find a "win" in whichever metric happened to look good.
- Open rate is the traditional measure for subject lines, but it has become unreliable. Privacy features in some email apps load images automatically, registering opens that may not reflect a real person reading. Treat open-based results with caution.
- Click rate (clicks divided by emails delivered) is usually a more dependable signal.
- Conversions, such as purchases, bookings or replies, are what you really care about, but they are rarer, so you need larger groups to detect differences.
Also keep an eye on unsubscribes and spam complaints. A subject line that wins on clicks but doubles complaints is not a winner. Mailbox providers watch complaint rates closely; see the sender guidelines from Google and Yahoo.
Sample size and why small lists struggle
The biggest trap in email testing is declaring a winner from too little data. Suppose each version goes to 200 people. Version A gets 8 clicks and version B gets 12. B looks 50 percent better, but with numbers this small, that gap could easily be random chance. Send the same two emails again and the result could flip.
A few rules of thumb:
- The smaller the difference you hope to detect, the more people you need. Detecting a small improvement can require many thousands of recipients per version.
- Rare outcomes need bigger groups. Testing for purchases needs far more recipients than testing for clicks.
- Use your tool's significance indicator if it has one, and understand what it is telling you. A result described as "statistically significant" means the difference is unlikely to be pure chance under the test's assumptions; it does not mean the difference is large or important.
If you want to understand the statistics behind comparing two proportions, the NIST/SEMATECH e-Handbook of Statistical Methods is a free, authoritative reference. Many free online sample-size calculators also exist for A/B tests.
Strategies for small lists
- Test bold changes, not tweaks. Big differences are easier to detect.
- Send the test to your whole list, split 50/50, rather than to a small sample first.
- Repeat tests. If the same idea wins across several sends, you can trust it more than a single result.
- Treat results as hints. Combine them with replies and customer conversations.
Avoiding false winners
- Do not stop the test early because one version pulls ahead in the first hour. Early results swing a lot, and people in different time zones open at different times. Decide the test length in advance.
- Allow enough time. Many tools let you choose how long to wait before picking a winner. A few hours may be too short for clicks and far too short for purchases.
- Keep everything else equal. Send both versions at the same time to equivalent groups.
- Watch for list differences. If your tool does not randomize properly, or one group contains more recent subscribers, the comparison is unfair.
- Beware novelty. A new style may win once simply because it is different, then fade.
Keep a test log
The real payoff from A/B testing is cumulative learning. Keep a simple log, in a spreadsheet or your CRM, with:
- Date, segment and list size
- Hypothesis (what you expected and why)
- The two versions
- Main metric and results for each version
- Your conclusion and confidence level
- What you will do differently next time
Over months this becomes a playbook for what works with your audience.
Getting started
A good first test for most businesses is the subject line of your regular newsletter, measured by clicks, sent 50/50 to the whole list. Once you have the habit, move on to offers and calls to action, which tend to affect revenue more directly.
Testing works best on a healthy, engaged list with clean data. If you are still building those foundations, start with email marketing basics and CRM data hygiene. And remember that every variation must still include the required unsubscribe and sender details; see email compliance basics.