How to A/B Test Personalized Images in Email (Without Rebuilding the Design)

You have a personalized image in a campaign and you want to know whether a different version performs better. Reasonably, you do not want to rebuild the whole thing twice, maintain two designs forever, and manually keep them in sync.

The good news is that this is easier than most people expect, because the job splits cleanly in two, and neither half involves redesigning anything.

Two jobs, two tools

The confusion in this topic comes from expecting one tool to do both halves. It doesn't, and it shouldn't.

Your email platform splits the audience. Klaviyo, Mailchimp, HubSpot, ActiveCampaign and the rest all have A/B testing built in. They decide who gets version A and who gets version B, they hold the split steady, and they report the results. This is their job and they are good at it.

Your image tool produces the variants. All it has to do is make a second version of the image cheap to create and cheap to swap in. That is the part that used to mean opening Photoshop.

So the question "which image tool has A/B testing?" is slightly the wrong question. The one worth asking is: how much work is it to produce variant B?

Making variant B without redesigning

With a URL-based personalized image, the design is identified by a parameter in the image URL. An OKZest image URL looks like this:

https://media.okzest.com/img?c=YOUR-COMPANY-ID&i=YOUR-DESIGN-ID&name={{first_name}}

To create a variant, you duplicate the design, change the one thing you want to test, and you get a new design ID. The variant URL is identical except for that ID:

Variant A:  https://media.okzest.com/img?c=YOUR-COMPANY-ID&i=DESIGN-A&name={{first_name}}
Variant B:  https://media.okzest.com/img?c=YOUR-COMPANY-ID&i=DESIGN-B&name={{first_name}}

Same company ID, same merge tags, same everything else. Your ESP puts URL A in version A of the email and URL B in version B, and the split it was already doing now covers the image as well.

Two consequences worth noticing:

The personalization keeps working in both. You are not choosing between "personalized" and "tested" - every recipient still gets their own name or data in whichever variant they receive.

You can test the image while holding the email constant. Because only the image URL differs, any difference in results is attributable to the image rather than to the subject line or the copy. That is a cleaner test than most people manage.

The mistake that quietly ruins the test

This one is specific to images that render when the email is opened, and it catches people out.

Do not edit a design while a test is running.

With open-time rendering, the image is generated at the moment the recipient opens the email, not at the moment you hit send. That is what allows the image to carry live data and to be corrected after sending. It also means that if you edit design A on Wednesday, every recipient who opens on Thursday sees the edited version - including people who received the email on Monday.

So a test where you tweaked variant A partway through is not a test of A versus B. It is a test of two versions of A and one of B, blended together in unknown proportions, and the result is unusable.

If you want to change something mid-flight, duplicate the design and treat it as a third variant with its own start date. Do not edit in place. The same caution applies to the design that is currently winning - leave it alone until the test is called.

What to test, in rough order of impact

Test one thing at a time. If you change three things and the number moves, you have learned that three things together did something, which is not knowledge you can reuse.

1. Whether the personalization is visible at a glance. The most common failure is a personalized image where the personalized part is small, low-contrast, or below the fold of the image. Testing a prominent treatment against a subtle one usually produces the biggest difference of anything on this list.

2. What data you personalize on. A first name is the default and it is the weakest signal, because everyone does it and recipients have stopped noticing. Testing a name against something the recipient actually cares about - their city, their plan, their usage, the item they left behind - is testing a genuinely different proposition.

3. The call to action inside the image. Not the button in the email, the text on the image itself. This is cheap to test and often overlooked.

4. Layout and composition. Where the personalized element sits relative to everything else.

5. Color and styling. Real but usually smaller than the four above, and it is where people tend to start because it is the easiest to have opinions about.

If you are on a first-name-only personalization today, start at 2. The gap between "Hi Sarah" and something that reflects what Sarah actually did is much larger than the gap between two shades of blue.

Measuring it without fooling yourself

This is where most image tests go wrong, and it has nothing to do with the images.

Do not judge the test on open rate. Apple Mail Privacy Protection prefetches images on the recipient's behalf, whether or not the person ever opens the email. A meaningful share of your list will therefore register an "open" that no human performed. Gmail proxies and caches images, which distorts things differently. Outlook blocks images by default for some recipients, which distorts them a third way.

The practical effect is that an image test measured by open rate is partly measuring which mail clients your recipients use. The variant that appears to win may simply have landed with more Apple Mail users.

Measure clicks instead, and ideally what happens after the click.

Click-through rate is the honest headline metric for an image test. A human decided to act.
Click-to-conversion is better still, because an image that drives curious clicks but no purchases is not actually winning.
Revenue per recipient, if you can attribute it, settles most arguments.

If you must look at opens, treat them as directional and never as the deciding number.

How long to run it

Long enough to be sure the difference is real, which is usually longer than people want.

The honest answer is that it depends on your baseline click rate and the size of the difference you are trying to detect - a small lift on a low-CTR list needs a much bigger sample than a large lift on a high-CTR one. Most ESPs have a significance calculator built into their A/B reporting, and it is worth trusting over your own eyes.

Some practical guidance regardless of the arithmetic:

Do not call it early. The first few hundred opens will show a dramatic difference that evaporates. This is normal and it means nothing.

Run through at least one full weekly cycle if your audience behaves differently on weekdays and weekends, which most B2B audiences do.

Be honest about what a null result means. If A and B come out level, that is a finding - it tells you that dimension does not matter to this audience, and you should test something else rather than testing shades of the same thing.

A small list may not be able to run this test at all. If you send to two thousand people with a 2% click rate, you have forty clicks to split between two variants, and no result you get from that is trustworthy. In that situation you are better off making one well-reasoned change and watching the trend over several sends than pretending to run an experiment.

What not to bother testing

Whether to personalize at all, if you have never done it. Just do it. You will learn more from testing two personalized versions against each other than from re-litigating the basic decision.

Tiny variations. Two words in a caption, a five-pixel shift. The effect is smaller than the noise, and you will spend a fortnight learning nothing.

Things you cannot act on. If you would not change your process based on either outcome, the test is entertainment.

Frequently asked questions

What's the easiest way to A/B test personalized image variations in email? Duplicate the design, change one element, and swap the design ID in the image URL for the test variant. Your email platform's existing A/B testing splits the audience - you only need to supply the second image URL. No redesign is required, and personalization continues to work in both variants.

Do I need an image tool with A/B testing built in? No. Your email service provider already splits the audience and reports the results. The image tool's job is to make producing the second variant quick, which a URL-based system does by changing a single parameter.

Can I test personalized images without a developer? Yes. Duplicating a design and copying an image URL are both editor tasks. A developer is only needed if you are generating images programmatically from your own system.

Why shouldn't I measure image tests by open rate? Because Apple Mail Privacy Protection prefetches images automatically, inflating opens for recipients who never read the email, and Gmail and Outlook each handle images differently again. An open-rate test partly measures mail client distribution rather than image performance. Use click-through rate instead.

Can I change the design while the test is running? No. Images that render when the email is opened will show the edited version to everyone who opens after the change, including recipients who were sent the original. That contaminates the test. Duplicate the design and run it as a new variant instead.

How many recipients do I need? Enough to get a meaningful number of clicks in each variant, which depends on your click rate and the size of the difference you are looking for. Use your ESP's significance calculator. If you are working with a few dozen clicks in total, the test will not tell you anything reliable.

What should I test first? Whether the personalized element is prominent enough to notice, followed by what data you are personalizing on. Both tend to produce larger differences than styling changes.

Can I run this across more than two variants? Yes, if your list is large enough. Each additional variant splits the audience further, so a three-way test needs substantially more volume than a two-way one to reach the same confidence.

Running your first one

The setup is smaller than it sounds: duplicate a design, change one element, put the second URL into your ESP's B variant, and leave it alone for a week.

OKZest generates personalized images from a URL, so producing a test variant means duplicating a design and swapping one ID - the merge tags, the data and the rest of the email stay exactly as they are. There is no per-variant design work and no rebuild.

Create a free account, no credit card required.