Optimizely

Overview

Heap automatically captures all active experiment variations as a property attached to every event on a given page. With Heap's Optimizely integration, you can answer questions like:

  • How does an A/B test impact long-term retention?
  • Do users exposed to an experiment convert faster than others?
  • Did a given experiment influence behavior that isn’t directly related to the functionality changed?

Setup

To get started, simply click CONNECT on the Optimizely Source page. Once connected, Heap will automatically capture all active experiments and exposed variations as a property on all events for a given page.

To ensure that Optimizely Pageview properties are properly captured, the Heap tag should always be deployed after the Optimizely tag, so that Optimizely has as much time to load as possible before Heap loads.

Data Format

Heap will automatically capture all active Optimizely experiments and attach a pageview-level property to all events on that page in the user's experience (including custom events). The format of the property follows the pattern of Optimizely: [Experiment Name] with the value of [Variation Name]. An example in this format might be: Optimizely: Home Page Test as the property name and Control as the value. Please refer to your experiment setup in Optimizely for more info on what these values will contain!

In practice, you'll see additional event-level properties on events where an active experiment is running. In the example image below, you can see a property entitled Optimizely: Collect and analyze - Homepage headline test with a value of Variation #1 included.

To use in analysis, simply apply a 'group by' clause to the analysis of interest and specify the event level property with the experiment property of interest to cut your analysis by the experiment exposed to the user at the time of the event!

Known Issues

Optimizely X

Privacy Settings: Mask descriptive names in project code and third-party integrations

Our integration attaches the active experiment name and variation names to events. If this option is enabled, experiment variation names aren't accessible by the integration and properties will not get sent to Heap. Ensure this option is disabled in your account by going to your Optimizely settings, then select "Privacy", and uncheck the checkbox for "Mask descriptive names in project code and third-party integrations".

Loss of Experiment Metadata

When using Optimizely with asynchronous loading, you may have chosen to load their tag using the asynchronous keyword in the script tag. For example:

<script src="//cdn.optimizely.com/js/1234567890.js" async=""></script>

In this case, the tag loads in the background as the rest of the page loads, which can cause screen flicker if the page loads faster than the tag and any variations that need to be applied. This particular method of deployment is explicitly not recommended by Optimizely. Google Tag Manager only allows for asynchronous tag deployment, which is one of the reasons why Optimizely recommends against using tag managers. If the tag is deployed in this manner, it is possible that Heap pageview data will be captured before the Optimizely metadata is loaded, which results in the loss of the contextual experiment information required to associate Heap auto-captured events with a particular experiment.

Other tag managers, such as Tealium, have the capability to deploy tags synchronously. This is the recommended way to deploy an Optimizely tag through a tag manager.

Optimizely offers the ability to load variations and custom code asynchronously. This can be configured within the Optimizely UI:

When using these options, the Optimizely tag should be deployed synchronously:

<script src="//cdn.optimizely.com/js/1234567890.js"></script>

The variations and custom code, however, will be loaded after the initial experiment metadata is loaded. Optimizely recommends only using asynchronous variation loading below-the-fold or for modals that will be closed upon page load. This will allow you to optimize your page load times while avoiding screen flicker. Additionally, because the experiment metadata is loaded synchronously, Heap will receive that data in the pageview, so long as the Heap tag is deployed in a lower position than the Optimizely tag. In this case, you can safely use asynchronous variation and custom code loading without worrying about loss of experiment metadata.