Headline Testing deep dive

Overview:

The Chartbeat Headline Testing tool allows homepage editors and content producers to actually evaluate how successful various headlines are at driving traffic to their articles, and actually help editorial teams get better at writing headlines that will attract visitors and hold their attention.
 
Chartbeat uses a multivariate approach to running live headline tests to determine if one headline is more successful at attracting interest from visitors (clicks) and holding the visitors’ attention for at least 15 seconds.
 
The multi-armed bandit approach, called Thompson Sampling, is different than other multivariate testing methodologies in that it exploits the benefits of the winning headline during the test. Essentially when a headline begins to perform better, Chartbeat will ‘play’ that headline more frequently.
 
ehtdeepdive.png
 
To read more about Thompson Sampling, check out this methodology report from our Data Science team. 

Are you a fit for Headline Testing?

There are a couple of technical prerequisites for Chartbeat's Headline Testing tool, so make sure that your site has cookies enabled, doesn't use virtual page changes, and has an average peak of 200 homepage concurrents. 
 
Check out this doc for more on implementation. 

How Do you Create an Experiment?

Once you have the Headline Testing tool installed, you'll select one of your headlines and enter as many headlines as you want to test against each other -- there's no limit but in order to have the experiment run efficiently we recommend 3 or 4. 

Once the headline test starts, it will run as many trials as it takes for one of the headlines to pull ahead in terms of popularity, until it reaches a play percentage of 95% and "wins". The length of an experiment depends on the volume of traffic to your homepage, the number of headlines you're testing, and the extent to which one headline is actually better than the other(s).

You can also access the Test Archives to get the full picture of all running, complete, and stopped experiments. 

Do you have any Tips for Headline Testing?

  1. Make sure that your headlines are different -- if there are only slight variations in your variants you're not going to get very good results, so don't be afraid to make them as different as possible. 
  2. Remember that your homepage audience is totally different than your social media audience. Don't assume that a headline that does well on your homepage will also do well at attracting visitors from Twitter or Facebook. 
  3. Test in higher traffic positions. Once you get comfortable with the tools, don’t be shy about trying headlines tests in the most prominent and highest Clicks per Minute regions of your home page. That’s where the most traffic benefit is and tests will run faster. 
  4. Don’t stop tests prematurely. It might be tempting to stop a test when the play percentage for a headline goes up quickly to 70 or 80%, but there will definitely be instances where headlines jump out to an early lead but then end up being inferior.
  5. Test from a hypothesis. Learnings and best practices will develop faster if you’re systematic about the experiments. What happens if you don’t give the whole story away in the headline? What happens if you use a casual tone? 

Looking for something else? We’re happy to help.