Real Networks Rhapsody

1
Multivariate Web Page Testing
Optimization SDMA November 2, 2005
2
Presentation Objectives

• Introduce You to Online Multivariate Testing
• Discuss When Multivariate Testing is Better than
  A/B Testing
• Outline the Process and Steps Required to Execute
  a Successful Multivariate Test
• Walk Through a Case Study

3
The Players
4
RealNetworks Consumer Products

• SuperPass
• Video News
• Music Downloads
• Premium Player
• Commercial Ad Free Radio
• 10 Free Songs/Mo
• 1 Free Game/Mo

• Rhapsody
• Purchased Listen.com in 2003
• Subscription Music Service
• All You Can Eat Pricing
• Discounted Downloads
• Manage Entire Catalog on PC
• Burn CDs, Create Playlists
• Personalized, Ad-Free Radio
• Music Recommendations

• Arcade/GamePass
• Casual and Arcade Games
• Free Pay per Download
• GamePass is Subscription Service

• MediaPlayer
• Premium Audio/Video
• Advanced CD Burning
• News Entertainment Content
• Music Store (.99 downloads)

5
Testing, Testing, A-B-C

• Traditional A/B Testing Tells You Which One
  Converts Better

But doesnt tell you why
6
Multivariate Testing Tells You Why

• Test Each Element Individually and in Conjunction
  with the Rest of the Page
• - Page Layouts
• Headlines and Subheads
• Location Color of Buttons
• Text Shape of Buttons
• Navigation yes or no
• Images size, quality, placement
• Bullets vs Paragraph
• Offers
• Analysis by Audience Segment

Instead of running hundreds of sequential A/B
tests to understand the impact, you run these
variables using experimental design principles.
7
Improvements Happen Through Iterations
8
Three Major Components to Multivariate Testing

• Methodology
• Determines the depth of the tests that can be run
  and the robustness of results.
• What is the experimental design behind the test.
• Technology
• Determines the breadth of testing and the ease of
  which it can be performed.
• Who hosts the page, what code is used to trigger
  the variables, how is the code integrated, etc.
• Experience
• Both the experience with testing and the
  experience behind the testing solution impacts
  the way the test is defined, produced, analyzed
  and communicated.

9
The Multivariate Test Process

• Planning

Development
Execution
Roll Out
Business Objectives
Pre-Wave Test
Move Pages to Full Production
Creative Development
Wave One
Test Plan Creation
Regression Analysis
Send Full Traffic
Technology Review
Wave Two
Regression Analysis
Confirmation Metrics
Matrix Creation
Wave Three
Template Creation
Regression Analysis
Case Study Presentation
Traffic Plan
Confirmation
Regression Analysis
10
Rhapsody Landing Page Case Study
Quiz How much better did this landing page work
than the control?
11
Landing Page Testing Objectives

• Create a landing page to boost click to trial
  conversions by at least 20.
• Understand which elements of the landing page are
  the most important in terms of driving results.
• Understand the differences between the behavior
  profile of the search audiences online music,
  artists and genres.
• Try a third party testing platform (Optimost) to
  understand the value and performance.

12
Business Challenge

• The Pay Per Click (PPC) Search Program had grown
  200 since previous summer in terms of Rhapsody
  trials while decreasing CPA 20 through
  increasing terms, managing bids more effectively
  and optimizing copy.
• The next step to growing the program was to
  improve the landing pages we used for search and
  affiliates.
• We needed to test 1,000s of variable combinations
  to get a significantly better landing page based
  on a click to trial conversion rate.

13
The Audience

• Traffic to the test pages came from two search
  publishers Google and Yahoo!
• Three major groups of terms pointed to three
  primary templates Music, Artists Genres
• A subset of groups were within each primary
• Music Download, MP3, Rhapsody, Radio, Music
• Artists Beyonce, Eminem, Kenny Chesney, etc.
• Genres Rock/Pop, RB/Soul, Rap/Hip-Hop, etc.
• We used a source code structure to send the right
  keyword group to the right template.
• ?srcppc_gg_artist

14
The Control Page Elements Tested
Top Navigation
Headline
Primary Image
Offer Copy
Button Color and Call to Action
Supporting Points

• Additional test items
• Page Layout
• Flash Tour
• Artist List
• Screenshots
• Move Proof Points Above the Fold
• Separate Offer Box

15
Example of a Test Matrix (first two rows only)

• Each value in each variable gets its own cell
• Include the text or the absolute link to the
  creative
• Go through it one more time with executives

16
Pages Optimosted Artist, Online and Genre
Landing Pages

• Test conducted from 5/12/04 to 07/01/04
• 5 layout templates
• 23 variables tested
• 124 values tested
• 1,093,680 template permutations analyzed

Control
17
Testing Methodology

• Divided each landing page layout into buckets
• Assign up to 7 variables to each bucket.
• Tested in three waves
• Wave 1 Focused on same variables with different
  page layouts AND a single new layout with extreme
  variables to discern which sections created the
  biggest differences.
• Wave 2 Focused on the winning layout vs control
  and additional values for variables
• Wave 3 Narrowed down to top performing values
  for each variable in validation wave against
  control.
• Statistical significance was targeted at the 95
  level inter-wave
• Success metrics were defined at two levels
• Order Path Entry Rate (the first page after the
  landing page) Used when not enough traffic made
  it to the trial thank you page.
• Trial Rate reaching the trial thank you page
  the PRIMARY success event.

18
Optimost Reporting Interface
Browser based reporting shows avgs in almost real
time include clicks, conversions, creative and
traffic Inter-wave reports are more succinct and
detailed with specific wins/losses and
recommendations
19
Topline Result Summary

• Winning creatives increased conversions to Order
  Path Entry pages by over 160 on all 3 pages
  (Online Music, Genre, Artist)
• Winning creatives increased actual registrations
  on Online Music page by 98.4 (increases on other
  pages were also very large, but sample size was
  too small to be significant)
• Winning elements were very consistent across
  Music, Online, and Genre pages
• Offer, Call-To-Action (text on the button),
  Testimonial Placement and Core Bullet Area are
  particularly critical to improvements in
  conversion

20
Winning layout were Significantly Better than
control and alternate layouts.
Vs.
Control Layout
Template Baseline Layouts
Alternate Layouts
21
Over 600,000 Songs and Ultimate Digital Music
Experience outperformed the control
Vs.
Control
22
Multivariate Testing Lessons Learned

• Get input from IT, Creative, Execs, Marketing
  early on to gain support and ideas for what
  should be tested.
• Test the widest conceivable options as well as
  the shades of gray.
• Make sure your test traffic is the same as your
  live traffic so that the results are consistent
  between the two.
• Make sure you have enough traffic to support the
  amount of testing you want.
• Think about starting smaller and building up to
  larger tests.
• Control for important elements like page weight
  in addition to the various messaging you want to
  test.
• Dont be afraid to test audience segments
• Dont be afraid to test shopping carts, forms or
  rich media content like flash or navigation
  elements.
• Be prepared to learn what doesnt work as well as
  what does especially in the first wave.
• Report back to the wider group with a case study,
  lessons learned and next testing ideas so that
  the lessons can be built into other
  products/programs.
• Keep a journal of tests and lessons learned.
  Centralize the information.

23
So, When Should You Multivariate Test?

• If you have a lot of ideas or questions on how to
  sell a product or service
• If you want to understand the relationship
  between variables as well as which ones perform
  better
• If you need results that show the best performing
  combination of elements sooner than later (vs a
  series of A/B testing)
• If you have a significant stream of traffic - the
  more traffic, the more variables and values you
  can test sooner.
• If you want see performance by segment the
  analytical and reporting tool must support this.

Multivariate Testing is NOT Always the
Answer. Use it as an Additional Tool in Your
Testing Kit.
24
More Information

• www.Real.com/Rhapsody
• Try Rhapsody for Free 14 Day Trial
• www.Real.com/SuperPass
• Try SuperPass for Free 14 Day Trial
• www.BrandDigital.net
• Workshops on Search Engine Marketing, Conversion
  and Customer Acquisition
• www.Optimost.com
• Multivariate testing platform, case studies, info
• www.PeakSystems.com
• Web creative and hosting