Test one thing at a time in cold email

What it does

Most cold email tests are noise. Two versions go out to 200 people each, one gets 6 replies and the other 9, and the “winner” becomes the new standard. At those numbers, a gap that size comes up by chance about 4 times in 10. Test one thing at a time, on big enough batches, and check whether the gap is real before you change anything.

What you need

• A sending tool that can split a campaign into versions
• A list big enough to split
• A spreadsheet

How it works

1. 1

   Pick one thing to test

   The first line, the question, the PS or the subject. Just one. If you change two things and it wins, you won’t know which one did it.

2. 2

   Write down what you expect before you send

   “The version with the PS gets more positive replies.” Measure positive replies, not opens. Opens aren’t reliable: some email apps load the tracking pixel whether or not anyone reads the email.

3. 3

   Split the list at random, and send both at once

   Same list, same days, same inboxes. Your sending tool’s A/B feature does the split. Never test one version this week and the other next week.

4. 4

   Check whether the gap is bigger than chance

   Google Sheets: paste into A1, swap in your numbers, and format D4 as a percent

   	Sent	Positive replies	Reply rate
   Version A	500	10	=C2/B2
   Version B	500	15	=C3/B3
   Chance a gap this big is luck			=2*(1-NORM.S.DIST(ABS(D2-D3)/SQRT(((C2+C3)/(B2+B3))*(1-(C2+C3)/(B2+B3))*(1/B2+1/B3)),TRUE))

   Under 5%, the gap is probably real. Above that, you don’t have a winner yet: keep both versions sending. As a guide from the standard sample-size math, telling a 2% reply rate from a 3% one with confidence takes about 3,800 sends per version, not a few hundred.

5. 5

   Keep the winner, then test the next thing

   Log every test: what changed, the sends, the replies, the chance figure and what you decided. If a winner surprises you, run it once more before it goes into every campaign.

What good looks like

A log of tests where every change you kept beat the old version by more than chance.

Mistakes to skip

• Testing on opens.
• Calling a winner after a few hundred sends.
• Changing the list and the words at the same time.
• Sending each version in a different week, or from different inboxes.