A/B testing in Campaigns

[Emily_Schickli]

Hello! I want to A/B test different email sequences in Campaigns to see which order of my messaging is the most effective for my onboarding sequence. I don’t see any help documentation around that. This is a pretty standard thing to want to do, so I would be surprised and disappointed if Keap Pro doesn’t allow that. Anyone do this successfully?

I wonder if I can use a decision diamond to test different campaign sequences since there doesn’t seem to be a built in % allocation switch function. I would imagine that I would either:

A) Apply no tags as rules, but not sure if Keap would split the audience or just send everyone through the first sequence in the list or

B ) Create a Zap that would randomly tag my new contacts with either “A” or “B” and then I would set up rules in the decision diamond based on that. Not sure if this is possible in Zapier.

Would appreciate any suggestions!

Best,
Emily

[Andy_Wroe]

So you are right that sending them down two sequences will be necessary. Unlike broadcast there is no inbuilt A/B testing.

If you tried option A, everyone would go into BOTH sequences.

So you are right, you need to do something like B. What you need to decide on is how you are going to split the data up. You haven’t said how big the list is, so it is hard to give you the best way of doing it. Lets say you have 1000 contacts you want to AB test.

I would sort by first name (as that’s pretty random) then using the grid view, you can have 500 contacts on page one.

Now unfortunately select all will get page 2 as well, so that’s a lot of clicks!

If you are confident in using the system and Excel then there is another method. Downloading your data, applying a tag in EXCEL and reloading it via Modify in Clean Up Your Data

Just be careful when you use this, I would ONLY be bringing ID and Tag back in to modify.

Hope that helps

Andy

[Emily_Schickli]

Thank you, Andy! That’s very helpful and will likely solve the problem when I want to test with an existing list. I’d like to run the test of two flows on my welcome / onboarding series for new users coming in through an ad, however.

As a result, I wouldn’t be able to do this as people are coming in at odd intervals constantly. Any suggestions here? One thought would be to split at the ad level, running two simultaneously with different url destinations to different lead magnet funnels and then tag users based on which one they came through (keeping the ads, landing pages, etc. exactly the same). Then have them go through two different email flows based on the tags they are assigned.

This flow seems like a lot of extra work though, so definitely open to other ideas!

Thanks so much for your help.

[Andy_Wroe]

What you have said above is totally correct!

So what you COULD do is use Zapier.

So contact enters the campaign, applies a tag that says fire Zap.

In the Zap you use the tag to grab the contact, you then use the Formatter tool.

In there choose Number - Spreadsheet style and enter the function Rand()

The you have another Formatter tool, again Number but this time Perform Maths Operation.

Take the value you just got in Rand and Multiply it by 100.

Ok so NOW in the next step put that into a custom field in Zapier. I’d also apply a tag to trigger the send of the email, that way you don’t have to guess the delay.

You can then use a decision diamond using 50+ and 0-49 as the A/B Test

Hope that helps

Andy

[Emily_Schickli]

Thank you! I’ll give it a shot – may have questions as follow-up when I do!

Best,
Emily

[Emily_Schickli]

Hi Andy! Thank you so much, I’ve tried the first few steps without issue and have my random number x 100, but I’m not sure how to add a custom field in Zapier. What function / app do I use for that? Is it in the actual Zap flow?

Thank you!

[Emily_Schickli]

Hi Andy, I’m also wondering how I would then add the custom random number to the contact itself so that I can pull it into a decision diamond in Keap. Or, would I assign in the Zap tag A or B rather than the random number and then use A or B in the Zap?

Would like to apply the number as it’ll be more scalable though for multivariate testing.

Thanks again for all your help!

[Emily_Schickli]

Hi Andy – sorry for all replies. But I think I may be on to something here. I’m not sure how to add a custom field in Keap so that I can populate it with this new Zapier random number. I have an old field I can then add the output to and use that in the decision diamond, but I’m not sure how to create a new one and name it in Keap.

[Emily_Schickli]

Okay last question for you before you respond… I also see in the decision diamond that I don’t seem to be able to do number-based logic, so greater than or less than. I only see these options below. Is that because of the way this custom field was set up? (I didn’t set it up and still am not sure how to).

[Andy_Wroe]

You walk away for 5 minutes…

Ok, so in Keap if you go to ANY contact record click edit on details, scroll to the bottom of the page you can create custom fields, you will want Number format.

Other than that I think you have it! Make sure you are putting the email back in and dedupe on email, in that final step in Zapier.

All the best

Andy

[Emily_Schickli]

Thank you so much, Andy! This was all super helpful. I’m testing out the workaround now REALLY appreciate all of your help.

Best,
Emily

[Gilad_Shir]

@Andy_Wroe How about a direct solution for A/B testing in campaigns?

To me it seems the bread and butter of a robust marketing automation system… Don’t you agree?