Tag Archives: Optimize

Lessons Learned: Testing and Optimization Tales from the Field

Max van der Heijden is a user experience and conversion specialist at Google who works with companies across Europe, the Middle East, and Africa. Max shares his thoughts about how companies can build a culture of growth and experimentation.


How many times have you launched new features or page designs on your website without testing first?

In an ideal world, companies should test everything before rolling out site changes. But some websites have too little traffic to generate credible results from experiments, and some bugs should just be fixed if they prevent users from achieving their goal. At the very least, analyze your analytics data and use qualitative methods such as user tests and surveys to validate any improvement ideas you have before implementing. If you have the traffic volume: Test!

I’m part of a team at Google that works with advertisers to identify opportunities for improving website user experiences through experiments and testing roadmaps. When our team of UX specialists begins consulting with a new company, the first three things I tell them are:

  1. The possibilities for improvement are enormous. Even if an experiment increases your conversion rate by “just 5%,” you can calculate the positive effect on your revenue.
  2. What works for one may not work for all. No matter how many times we have seen recommendations or “best practices” work on other — maybe even similar — websites, that does not mean it will work for your users or your business.
  3. Expect failures — and learn from them. Testing takes time, and it's hard to know which tests will pay off. Embrace failures and the lessons learned from them.

Making the switch from “get-it-live” mode to a test-and-learn mindset takes time and effort. Leading companies are building a culture of growth: one where people focus on using data and testing to optimize the customer experience day by day. Below are some of the key lessons learned as we work with teams embracing this growth mindset.

Get top-level support

When we first talk with customers, we insist a decision-maker attend our meetings. If there's no support from the top, all of our testing ideas could end up on the shelf collecting dust. Obviously, the marketing executive or CEO won’t have an a-ha moment if you frame testing as a way to improve conversions. The trick is to show how testing impacts a business goal, such as revenue or, better yet, profit. Then the decision-maker will have an ohhh moment: As in, “Ohhh, I knew this was important, but I didn’t think about how a small change could have such a big impact on our bottom line.”

Top-level support will help you get the resources you need and unlock the potential of people already working on experiments. The trend we see is typically one or two persons who start doing the optimizations. They are usually mid-level designers or data analysts who have an affinity for conversion rate optimization, but are often working in a silo.

On the other end of the spectrum, we see companies that have fully bought into the power of experimentation. Multiple customers even have a group of product managers who work on projects with a group of specialists, including a data scientist, copywriter, designer, and even a design psychologist.

Tip: Look for these three types of people to jumpstart a culture of growth in your organization.

Prioritize, prioritize, prioritize

You can't test every idea at once. And prioritization should not be a guessing game.

When we surveyed a group of our EMEA advertisers at a conversion rate optimization event, 38% of the respondents said they use their gut or instinct to prioritize, while 14% allow the HiPPO (highest paid person’s opinion) to call the shots.1 Instead, try using a framework that takes into account past lessons learned and resource requirements.

Map test ideas in a speed-versus-impact grid, and prioritize experiments that are quick to launch and likely to have the biggest impact. Keeping track of all prior test results is another way to ensure past learnings come into play when faced with a HiPPO.

Tip: Start with ideas that will be simple to test and look like they could have the biggest potential impact.


Turn fairweather fans into engaged experimenters

Over time, as you share testing insights and achieve a few wins, more people will jump on board and you’ll need to train people on a repeatable testing framework.

Testing is part of a cycle: What does the data tell you? Did the experiment succeed or fail for every user, or just for a specific segment of users? Analyze your test results, especially from failed experiments, and use those insights to improve the customer experience across your touchpoints. And then conduct another test.

Just as important: How do you keep people excited and engaged in the process? Try using a shared document to invite everyone to submit their improvement suggestions for your website or app. You can even add gamification to this by keeping score of the most impactful ideas. Or, have people guess which test variation will win before you run the test. When you share the results, recognize or reward people who correctly predicted the winner.
Tip: Three ways to get your team engaged with testing and optimization

Feel good about failures

By its very nature, experimentation involves a lot of failure. A typical website might have 10 or 100 or even 1,000 things to test, but it might be that only a small percentage of those tests lead to significant, positive results. Of course, if that one winner leads to a 5% or 10% improvement in conversions, the impact on revenue can be enormous.

When we surveyed EMEA advertisers at one of our events, we found that companies running one to two tests a month had a 51% success rate. But for respondents who said they ran more than 21 tests a month, the success rate decreased to 17%.2

In the beginning, it’s easier to spot areas for improvement and “low-hanging fruit.” The more experiments you run, the more you’ll be focusing on smaller and smaller things. Then, the more you test, the less “successful” you will be. "Our test success rate is about 10%," says Jesse Nichols, Head of Growth at Nest. "But we learn something from all our tests."

Download the guide How to Build a Culture of Growth to learn more about best practices for testing and optimization.

1-2 Source: Conversions@Google 2016 - State of CRO event attendee survey, 145 respondents, EMEA, September 2016.

Posted by Max van der Heijden, user experience and conversion specialist at Google, EMEA

What does a good website test look like? The essential elements of testing

"Test! Test! Test!" We've all heard this advice for building a better website. Testing is the heart of creating a culture of growth ― a culture where everyone on your team is ready to gather and act on data to make the customer experience better day by day.

But how do you run a good test? Is it just a matter of finding something you're not sure about and switching it around, like changing a blue "Buy now" button for a red one? It depends: Did you decide to test that button based on analytics, or was it a wild guess?

Assuming the former, a good test also means that even if it fails, you’ve still learned something. A bad test may make your website performance worse than before, but it’s even worse if you don’t take those learnings into account in the future.

The key to running good tests is to establish a testing framework that fits your company.

Join us for a live webinar on Thursday, March 9, as Krista Seiden, Google Analytics Advocate, and Jesse Nichols, Head of Growth at Nest, share a six-step framework for testing and building better websites.

Frameworks vary from business to business, but most include three key ideas:

Start with an insight and a hypothesis.
A random "I wonder what would happen if …" is not a great start for a successful test. A better way to start is by reviewing your data. Look for things that stand out: things that are working unusually well or unusually badly.

Once you have an insight in hand, develop a hypothesis about it: Why is that element performing so well (or so badly)? What is the experience of users as they encounter it? If it's good, how might you replicate it elsewhere? If it's bad, how might you improve it? This hypothesis is the starting point of your test.

For example, if you notice that your mobile conversion rate was less than on desktop, you might run tests to help you improve the mobile shopping or checkout experience. The team at The Motley Fool found that email campaigns were successfully driving visitors to the newsletter order page, but they weren’t seeing the conversions. That led them to experiment on how to streamline the user experience.

Come up with a lot of small ideas.
Think about all the ways you could test your hypothesis. Be small-c creative: You don't have to re-invent the call-to-action button, for instance, but you should be willing to test some new ideas that are bold or unusual. Switching your call-to-action text from "Sign up now" to "Sign up today" may be worth testing, but experimenting with "Give us a try" may give you a broader perspective.

When in doubt, keep it simple. It's better to start with lots of small incremental tests, not a few massive changes. You'll be surprised how much difference one small tweak can make. (Get inspiration for your experiments here.)

Go for simple and powerful.
You can't test every idea at once. So start with the hypotheses that will be easy to test and make the biggest potential impact. It may take less time and fewer resources to start by testing one CTA button to show incremental improvement in conversion rates. Or, you may consider taking more time to test a new page design.

It may help to think in terms of a speed-versus-impact grid like this. You don't want quiet turtles; the items you're looking for are those potential noisy rabbits.


The best place to begin a rabbit hunt is close to the end of your user flow. "Start testing near the conversion point if you can," says Jesse Nichols, Head of Growth at Nest. “The further you go from the conversion point, the harder it gets to have a test that really rocks — where the ripple effect can carry all the way through to impact the conversion rate,” says Jesse.

Stick with it
A final key: Test in a regular and repeatable way. Establish an approach and use it every time, so you can make apples-to-apples comparisons of results and learn as you go.

A clear and sturdy framework like this will go a long way toward making your team comfortable with testing — and keeping them on the right track as they do.

Download the eBook How to Build a Culture of Growth to learn more about best practices for testing and optimization.

Posted by Tiffany Siu, Product Marketing Manager, Google Optimize 360

Why Building a Culture of Optimization Improves the Customer Experience

How can we be more useful to our customers today?

That's the simple question that drives any marketing organization focused on testing, improvement, and growth.

But answering the question is not always so simple in our data-rich world. The old challenge of gathering enough data has been replaced by a new one: gleaning insights from the mountains of data we’ve collected — and taking action.

In response to this flood of data, many of today's most successful businesses have turned to a new approach: building what's called a culture of growth and optimization.

This growth-minded culture is one where everyone is ready to:
  • Test everything 
  • Value data over opinion 
  • Keep testing and learning, even from failures 
Most companies have a few people who are optimizers by nature, interest, or experience. Some may even have a “growth team.” But what really moves the dial is when everyone in the company is on board and embraces the importance of testing, measuring, and improving the customer experience across all touchpoints.
"We refuse to believe that our customers’ experiences should be limited by our resources." - Andrew Duffle, Director of Analytics, APMEX
Why should marketers care?
Because they'll be leading the revolution. 86% of CMOs and senior marketing executives believe they will own the end-to-end customer experience by 2020, according to a recent survey from the Economist Intelligence Unit.1 And a culture of growth and optimization offers an excellent path to major gains in those experiences.

As testing and optimization proves itself, it tends to generate higher-level investments of support, talent, and resources. The payoff arrives in the form of more visitors, more sales, happier customers and a healthier bottom line.

If you're curious about building a culture of optimization in your marketing organization, register for our Nov. 10 webinar, Get Better Every Day: Build a Marketing Culture of Testing and Optimization.

This webinar will cover:
  • The critical elements of a culture of optimization 
  • Tips for building that culture in your own company 
  • A case study discussion with Andrew Duffle, Director of Analytics at APMEX, a retailer that boosted revenues with continuous testing and optimization 
This kind of culture doesn't happen by command, but it’s also simple to start building.

We look forward to sharing tips on how you can get started. Happy optimizing!


  1. The Economist Intelligence Unit, "The Path to 2020: Marketers Seize the Customer Experience." Survey and a series of in-depth interviews with senior executives. Survey base: 499 CMOs and senior marketing executives, global, 2016.

Posted by Jon Mesh, Product Manager, Google Optimize and Google Optimize 360

How to Apply Holiday Shopping Insights to Your Analytics Strategies

The Year of the Supershopper
We all have that friend — the one who somehow knows the latest brands, the season’s must-have products, and where to find the best deals at the snap of a finger. In years past, this friend was an enigma, making us wonder how does he or she do it?

Today, we can all be that friend. With the ability to instantly discover, research, and purchase, shoppers around the world are more informed and more efficient than ever before - they’ve transformed into supershoppers seemingly overnight.

But what defines supershoppers? And what does this mean for retailers trying to win them over this holiday season? Let’s find out.


They Keep Their Options Open
Last year, more than 50% of holiday shoppers said they were open to purchasing from new retailers1. This is especially true online. More than three-quarters of smartphone shoppers who usually go to the same physical stores when they shop for products are very open to new retailers and brands online2. Why? Mobile makes it easy to explore all of your options no matter when or where you’re shopping. In fact, after searching on Google, 76% of mobile shoppers have changed their mind about which retailer or brand to purchase3.
Mobile is Their Muse
It used to be that shoppers would thumb through catalogues or stare longingly at the holiday window displays, but mobile is now the super shopper’s go-to source for inspiration. Sixty-four percent of smartphone shoppers turn to mobile search for ideas about what to buy before heading into a store4. And 1 in 4 mobile video viewers in the U.S. have visited YouTube for help with a purchase decision while they were at a store or visiting a store's website5.

But shoppers aren’t only making purchase decisions, they’re discovering new brands and products along the way: more than half of smartphone users have discovered a new company or product when conducting a search on their phones6.

They Want the Best - Not the Cheapest
In July we learned that shoppers are on the quest for the best — and this still rings true more than ever today. Last holiday, mobile searches related to “best gift” grew 70% year over year while mobile searches related to cheap or inexpensive gifts grew about 35%7. They’re also willing to do the research to the make the best decision: on YouTube, mobile watch time for product review videos has grown 60% year over year8.

But supershoppers don’t only want the best - they want personalized, unique, cool gifts. Mobile searches related to “unique gifts” grew more than 65% while mobile searches related to “cool gifts” grew a whopping 80%9.

Mobile is Their Door to the Store
Although more and more people are willing to buy on mobile, we know that mobile is still used predominantly as a door to the store. In fact, 76% of people who search for something nearby on their smartphone visit a related business within a day, and 28% of those searches result in a purchase10.

Once they’re inside your store, they expect the experience to be a seamless one: more than 40% of smartphone shoppers want retailers to automatically surface relevant information such as the location of the item in the store, a special deal or related products11.

They Shop ‘Til they Drop
Supershoppers live up to their name as the holiday season progresses. From November through mid-December we see online conversion rates increase across devices. Last year, on mobile alone, they jumped 30% on Black Friday and 50% on Cyber Monday when compared to November 112.



Tips to Using Analytics to Reach Today’s Shopper
Analytics is a critical marketing tool all year long. But that importance is amplified during the holiday season. Here are five ideas on how you can use analytics to get the most out of the holiday shopping season.

1. Understand which days drive the most sales — be there. In order to develop a successful holiday retail strategy, it’s important to first understand the days that drive the most sales for your business. Once you understand this, you can craft strategy — across mobile, desktop, and tablet — that optimizes your media and promotion not only for these days, but for the entire holiday season.
For example, adjust bids for auction-based media and use the lift in transaction rates over the average transaction rate as your bid multiplier. Use Smart Bidding in AdWords or DoubleClick Search to maximize your performance.

2. Get personal when engaging with window shoppers. The great thing about the holidays is that you can start shopping at any time — and then wait for discounts. Many consumers may already be browsing your site looking for gift ideas. This is an audience that may convert at a higher rate. Use Google Analytics to create remarketing lists including these shoppers, and then customize your marketing campaigns to better suit their needs.

But don’t stop there: Remember, supershoppers crave personalization. Test different variations of your website with this same audience to offer more customized experiences. Try Google Optimize, our free site testing and personalization tool, to get started.

3. Act fast: Create a culture of optimization. Your holiday marketing plans are set long before the holiday season starts. It’s very important to monitor your activities to ensure they’re driving business growth. And if they’re not you need to take action by immediately identifying areas of opportunity and improvement.

Google Analytics offers a number of ways to help simplify reporting, sift through data, and spotlight insights for you. Custom reports, dashboards and shortcuts all let you customize reporting so you spend less time looking for data and more time analyzing important information. Or better yet, try Google Data Studio and create a holiday shopping dashboard for different teams in your organization.

4. Analyze your performance against your competitors. Benchmarking your business against your competitors helps you see the big picture. If you’re a Google Analytics user, you can tailor this analysis and approach, using your own data and the data available in our benchmarking reports. The benefit of using those reports is that they will provide you a comparison between your data and your competitors, using characteristics such as website traffic, country and detailed vertical information.

5. Start developing your New Year marketing strategy. With so many new shoppers entering your stores, sites, and apps, data and analytics are critical to helping you convert first-time shoppers into long-term, loyal customers. Use your website data to create lists of first-time customers during the holidays, and deliver personalized communication across channels to build relationships. With Google Analytics you can create remarketing lists and easily connect with this audience.

Happy analyzing!

Posted by Julie Krueger, Retail Managing Director, Google and Casey Carey, Director of Analytics Marketing, Google


  1. Google/ Ipsos, Post Holiday Shopping Intentions Study - Total Shoppers Report, Jan 2016, Base: US Holiday Shoppers, n=1,500
  2. Google/Euromonitor International, Micro-Moments Survey, US, July 2016, Smartphone shoppers = 1000, Same store shoppers = 801
  3. Google/Euromonitor International, Micro-Moments Survey, US, July 2016, Smartphone shoppers = 1000
  4. Google/Euromonitor International, Micro-Moments Survey, US, July 2016, Smartphone shoppers = 1000
  5. Google-commissioned Ipsos Brand Building on Mobile Survey, U.S., December 2015 n=1000, 18-54 year olds
  6. Google/Ipsos, "Consumers in the Micro-Moment," Wave 3, U.S., n=1291 online smartphone users 18+, August 2015
  7. Source: Google Search Data, Apparel, Home & garden, Beauty & personal care, Computers & electronics, Gifts, Toys & games, Photo & video, Nov-Dec 2014 vs Nov-Dec 2015, United States
  8. YouTube data, U.S., Classification as a “Product Review" video was based on public data such as headlines and tags, and may not account for every such video available on YouTube, Nov - Dec 2014 and Nov - Dec 2015
  9. Google Search Data, Apparel, Home & garden, Beauty & personal care, Computers & electronics, Gifts, Toys & games, Photo & video, Nov-Dec 2014 vs Nov-Dec 2015, United States
  10. Google/Purchased Digital Diary: How Consumers Solve Their Needs in the Moment, May 2016, Representative sample of US Smartphone users = 1000. Local searchers = 634, Purchases = 1,140
  11. Google/Euromonitor International, Micro-Moments Survey, US, July 2016, Smartphone shoppers = 1000
  12. Google Analytics Shopping category data, Nov 1, 2015–December 14, 2015, United States