Marketing API Version

Lift study

This guide covers the steps needed for advertisers and developers to setup a new lift study.

A lift study allows the advertiser to create an experiment to measure the efficacy of their Facebook campaigns and/or determine what strategy is driving the most business impact.

When creating a lift study, a randomized test group (people that see ads) and control group (people that don't) are established. The advertiser can securely share conversion data from the campaign with Facebook via Conversion Pixels, Facebook Pixels, and/or Mobile App. Facebook will determine the additional lift generated from the campaign by comparing the number of conversions, people converting, and sales revenue (if available) in the test and control groups. The results will be available via Ads Manager.

Setting up a study

A lift study can be set up with one or more test groups—called “cells”. When you set up your study, Facebook randomizes the audience and assigns people into either the test or control group(s). After running the study, Facebook will calculate the difference between the test group(s) and control group(s) so that you can assess the impact of Facebook ads on business outcomes.

A study with a single test group lets you see how advertising on Facebook may lead to additional business. You can also set up a study with multiple test groups, which lets you determine what advertising approach works best for your audience.

For more details on the endpoint, refer to the Ad Study reference documentation

To properly set up a study, you will need to make a POST call to the following endpoint:

https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

You should provide the following information to create a new study:

  • The name of the study.
  • A brief description about the purpose of the study.
  • cooldown_start_time. This is the start of your pre-campaign quiet period. During this time, Facebook pauses all ads within the study to establish a quiet period for baseline; ads will resume once the study period starts. If a quiet is not needed for you study, you can set this value to be the same as the start_time.
  • The start_time of the campaign active period, when ads will start running. Study start time must be in the future.
  • The end_time of the campaign active period, when ads will stop running.
  • observation_end_time. This is the end of your post-campaign quiet period which began when the study period ends. Conversions that are passed to Facebook during this period will still attribute to the study. Ads with in the study will be paused during this period as well. If a quiet is not needed for your study, you can set this value to be the same as the end_time.
  • cells of the study to describe the test and control groups.
  • The objectives of the study. See Defining Study Objective section for more details.
  • confidence_level specifies the statistical confidence levels (defaults to 90%) for conversion objectives in Lift setup. Valid confidence levels are 0.8, 0.9, and 0.95.
  • viewers. You can share this study by providing a list of Facebook User IDs.
  • The type of study. For Conversion Lift, the type should be LIFT.

Test Group

You set up a test group to determine how many people will be exposed to your ads and how many people will not.

A test group must be created inline as you're setting up the study by passing a list of JSON objects to the cells parameter.

For more details on cells, refer to the Ad Study Cell reference documentation

A test group object contains the following information:

  • The name of the test group.
  • The description of the test group.
  • The treatment_percentage defines the people who will be exposed to the ads. The percentage may be a floating point value of up to two decimal places.
  • The control_percentage defines the holdout percentage of the people who will not see ads. The percentage may be a floating point value of up to two decimal places. The treatment and control percentages must sum to 100.
  • The list of ad entities, such as adaccounts or campaigns, that you want to study. All ads under the ad entities that are active during the study period will be included.

Here is an example of how to set up a lift study with one test group.

curl \
-F 'name="new study"' \
-F 'description="description of my study"' \
-F 'start_time=1435622400' \
-F 'end_time=1436918400' \
-F 'cooldown_start_time=1433116800' \
-F 'observation_end_time=1438300800' \
-F 'viewers=[<USER_ID1>, <USER_ID2>]' \
-F 'type=LIFT' \
-F 'cells=[{name:"test group",description:"description of my test group",treatment_percentage:89.75,control_percentage:10.25,adaccounts:[<ACCOUNT_ID1>,<ACCOUNT_ID2>]}]' \
-F 'objectives=[{name:"new objective",is_primary:true,type:"MAI",applications:[{id:<APP_ID>}]}]' \
-F 'confidence_level=0.9' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

You can do this to retrieve the test groups that are in the study.

curl -G \
-d 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_ID>/cells

You can update or modify cell information along with the threatment and control percentages at the study level by providing the cell ID in the cells field.

curl \
-F 'cells=[{id:<CELL_ID>,treatment_percentage:80,control_percentage:20}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_ID>

Once the study starts running, you cannot update the start_time and treatment_percentage of the cells. You also cannot remove the associated objects, such as adaccounts or campaigns, of the test groups. However, You can still update the end_time and observation_end_time to the future time if the study is not yet ended and add new associated objects to the test groups if needed.

If you want to run Reach and Frequency in conjunction with Lift measurement, you must set up a Lift study first and make sure the duration of the Reach and Frequency is within the duration of the Lift study.

You can also read all the studies that you have created from the ad_studies edge of your business.

curl -G \
-d 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

Setting up a study with multiple test groups

You can also set up a study with multiple test groups of Facebook users to measure incremental impact of different Facebook strategies, such as different targeting options, on business outcomes.

It is important to note that the control_percentage determines the holdout for each test group respective to the total population. For example, you have a study with two test groups—Test Group A is 50% treatment with 20% control and Test Group B is 20% treament with 10% control. This will result in ~28.6% (20%/70%) of the population in Test Group A to be control users and ~33.3% (10%/30%) of the population in Test Group B to be control users.

To set up a study with multiple test groups, you can provide a list of test groups objects in the cells parameter. The treatment and control percentages across the test groups must sum to 100. However, it can be less than 100 for some specific use cases. For example, when you have three test groups that are split evenly at 33%.

curl \
-F 'name="new study"' \
-F 'description="description of my study"' \
-F 'start_time=1435622400' \
-F 'end_time=1436918400' \
-F 'cooldown_start_time=1433116800' \
-F 'observation_end_time=1438300800' \
-F 'viewers=[<USER_ID1>, <USER_ID2>]' \
-F 'type=LIFT' \
-F 'cells=[{name:"group A",description:"description of group A",treatment_percentage:50,control_percentage:20,campaigns:[<CAMPAIGN_ID1>]},{name:"group B",description:"description of group B",treatment_percentage:20,control_percentage:10,campaigns:[<CAMPAIGN_ID2>]}]' \
-F 'objectives=[{name:"new objective",is_primary:true,type:"MAI",applications:[{id:<APP_ID>}]}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

You can update, add, and remove test groups in a study by doing so at the study level. To update an existing test group, refer to its ID in the test group object. To add a new test group, provide a new test group object. To remove a test group, simply omit it from the cells parameter when you update it.

curl \
-F 'cells=[{id:<CELL_ID1>,treatment_percentage:60,control_percentage:10},{name:"group C",description:"replacing group B",treatment_percentage:25,control_percentage:5,campaigns:[<CAMPAIGN_ID3>]}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_ID>

Defining Study Objective

Study objectives define what advertising objectives you want to measure and how you will pass the conversion data to Facebook. A lift study requires at least one objective for proper reporting.

NameDescriptionMeasurement Sources

SALES

Measure the lift in purchases. The reporting will contain dollar amount.

Conversion Pixels, Facebook Pixels, and Mobile App.

NONSALES

Measures the lift in non-purchase conversions.

Conversion Pixels, Facebook Pixels, and Mobile App.

MAI

Measure the lift in mobile application installs.

Mobile App

For SALES and NONSALES objectives, if Facebook Pixel or Mobile App is used as measurement sources, you must provide a list of the event names that you want to capture for the objective. This allows Facebook to report the result of the measurement based on these conversion events.

Measurement SourceEvent Names

Facebook Pixel

fb_pixel_view_content, fb_pixel_search, fb_pixel_add_to_cart, fb_pixel_add_to_wishlist, fb_pixel_initiate_checkout, fb_pixel_add_payment_info, fb_pixel_purchase, fb_pixel_lead, fb_pixel_complete_registration

Mobile App

fb_mobile_activate_app, fb_mobile_complete_registration, fb_mobile_content_view, fb_mobile_search, fb_mobile_rate, fb_mobile_tutorial_completion, fb_mobile_add_to_cart, fb_mobile_add_to_wishlist, fb_mobile_initiated_checkout, fb_mobile_add_payment_info, fb_mobile_purchase, fb_mobile_level_achieved, fb_mobile_achievement_unlocked, fb_mobile_spent_credits

For more details on objectives, refer to the Ad Study Objective reference documentation

An objective can be created inline by passing a list of JSON objects the objectives parameter when making a POST call to create a new study.

An objective object contains the following information:

  • The name of the objective.
  • A boolean, is_primary, to decide if this is the primary objective. A study can only have one primary objective.
  • The type of the objective: SALES, NONSALES, or MAI.
  • adspixels. A list of Facebook Pixel IDs along with the relevant list of event_names per id if applicable.
  • offsitepixels A list of Conversion Pixel IDs if applicable.
  • applications. A list of mobile applications along with the relevant list of event_names per id if applicable.

The objectives of a study cannot be modified after the study has begun.

For example, you can create and add a primary MAI objective to a study.

curl \
-F 'name="new study"' \
-F 'description="description of my study"' \
-F 'start_time=1435622400' \
-F 'end_time=1436918400' \
-F 'cooldown_start_time=1433116800' \
-F 'observation_end_time=1438300800' \
-F 'viewers=[<USER_ID1>, <USER_ID2>]' \
-F 'type=LIFT' \
-F 'cells=[{name:"test group",description:"description of my test group",treatment_percentage:90,control_percentage:10,adaccounts:[<ACCOUNT_ID1>,<ACCOUNT_ID2>]}]' \
-F 'objectives=[{name:"new objective",is_primary:true,type:"MAI",applications:[{id:<APP_ID>}]}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

You can also have multiple objectives per study. The result will be aggregated based on objectives. Here is an example of a study with multiple objectives—the MAI objective as seen previously and another SALES objective with Facebook Pixel and Application as measurement sources.

curl \
-F 'name="another study"' \
-F 'description="description of another study"' \
-F 'start_time=1435622400' \
-F 'end_time=1436918400' \
-F 'cooldown_start_time=1433116800' \
-F 'observation_end_time=1438300800' \
-F 'viewers=[<USER_ID1>, <USER_ID2>]' \
-F 'type=LIFT' \
-F 'cells=[{name:"test group",description:"description of my test group",treatment_percentage:90,control_percentage:10,adaccounts:[<ACCOUNT_ID1>,<ACCOUNT_ID2>]}]' \
-F 'objectives=[{name:"MAI objective",is_primary:true,type:"MAI",applications:[{id:<APP_ID1>},{id:<APP_ID2>}]},{name:"SALES objective",type:"SALES",applications:[{id:<APP_ID3>,event_names:["fb_mobile_purchase"]}],adspixels:[{id:<FB_PIXEL_ID>,event_names:["fb_pixel_purchase","fb_pixel_lead"]}]}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

You can update, add, and remove objectives in a study by doing so at the study level similar to modifying test groups. To update an existing objective, refer to its ID in the objectives object. To add a new objective, provide a new objective object. To remove an objective, simply omit it from the objectives parameter when you update it.

Here is an example of updating an objective's applications measurement sources and removing its adspixels measurement sources:

curl \
-F 'objectives=[{id:<OBJECTIVE_ID>,name:"new objective name",applications:[{id:<APP_ID>}],adspixels:[]}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_ID>

You can read the objectives that were created for the study.

curl -G \
-d 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_ID>/objectives

Reporting

Retrieving objectives

A study's objectives are defined during the study setup. See the setup guide on how to define your study's objectives

You can read the objectives that were created for a study by making a GET call to the study's objectives edge.

curl -G \
-d 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_ID>/objectives

For more details on objectives, refer to the Ad Study Objective reference documentation

Retrieving results

To retrieve results for an objective, you can make a GET call to the objective node by specifying results in the fields parameter.

curl -G \
-d 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_OBJECTIVE_ID>?fields=results

The resulting data is a stringified JSON object with the following default fields:

NameDescription

population.test

The number of people included in the test group of the experiment.

population.control

The number of people included in the control group of the experiment.

polutation.reached

The number of people in the test group that were reached by 1 or more ads associated with the lift study.

impressions

Number of billable ad impressions for ads associated with the lift study.

spend

The total amount spent up to the current point of your study.

frequency

The average number of times your ad was shown to each person in population.reached.

buyers.test

The number of buyers from the test group of the experiment.

buyers.control

The number of buyers from the control group of the experiment.

buyers.scaled

The number of buyers in the control group, factored-up by the ratio of population.test/population.control to produce a fair comparison for buyers.test.

buyers.incremental

The increased number of buyers as a result of running campaigns. Calculated as: buyers.test - buyers.scaled.

buyers.delta

The 90% confidence interval associated with the estimate for buyers.incremental.

buyers.pValue

The p-value associated with buyers.incremental.

buyers.isStatSig

1 = buyers.incremental is statistically significant at 90% confidence. 0 = buyers.incremental is not statistically significant at 90% confidence interval.

buyers.reached

The number of buyers reached by your ads.

buyers.reachedPercent

buyers.reached / buyers.test

buyers.baseline

buyers.reached - buyers.incremental

buyers.lift

buyers.incremental / buyers.baseline

conversions.test

The number of conversions from the test group of the experiment.

conversions.control

The number of conversions from the control group of the experiment.

conversions.scaled

The number of conversions in the control group, factored-up by the ratio of population.test/population.control to produce a fair comparison for conversions.test.

conversions.incremental

The increased number of conversions as a result of running campaigns. Calculated as: conversions.test - conversions.scaled.

conversions.delta

The 90% confidence interval associated with the estimate for conversions.incremental.

conversions.pValue

The p-value for conversions.incremental.

conversions.isStatSig

1 = conversions.incremental is statistically significant at 90% confidence. 0 = conversions.incremental is not statistically significant at 90% confidence interval.

conversions.reached

The number of conversions from buyers reached by your ads.

conversions.reachedPercent

conversions.reached / conversions.test

conversions.baseline

conversions.reached - conversions.incremental

conversions.lift

conversions.incremental / conversions.baseline

incrementalROAS

Incremental return on ad spend. Calculated as: sales.incremental / spend.

sales.test

The amount of sales from the test group of the experiment.

sales.control

The amount of sales from the control grooup of the experiment.

sales.scaled

The amount of sales in the control group, factored-up by the ratio of population.test/population.control to produce a fair comparison for sales.test.

sales.incremental

The increased amount of sales as a result of running campaigns. Calculated as: sales.test - sales.scaled.

sales.delta

The 90% confidence interval associated with the estimate for sales.incremental

sales.pValue

The p-value associated with sales.incremental.

sales.isStatSig

1 = sales.incremental is statistically significant at 90% confidence. 0 = sales.incremental is not statistically significant at 90% confidence interval.

sales.reached

The amount of sales from buyers reached by your ads.

sales.baseline

sales.reached - sales.incremental

sales.lift

sales.incremental / sales.baseline

Sample response shown as parsed JSON for ease of reading:

{
  "id": "<STUDY_OBJECTIVE_ID>",
  "results": [{
    "population.test": 1349292,
    "population.control": 336567,
    "polutation.reached": 738219,
    "impressions": 3349184,
    "spend": 23443.12,
    "frequency": 4.5368434028385,
    "buyers.test": 212300,
    "buyers.control": 50823,
    "buyers.scaled": 203748.63642603,
    "buyers.incremental": 8551.3635739689,
    "buyers.delta": 1536.450929913,
    "buyers.pValue": 0,
    "buyers.isStatSig": 1,
    "buyers.reached": 111199,
    "buyers.reachedPercent": 0.52378238341969,
    "buyers.baseline": 102647.63642603,
    "buyers.lift": 0.08330794426164,
    "conversions.test": 1473655,
    "conversions.control": 362552,
    "conversions.scaled": 1453465.4710177,
    "conversions.incremental": 20189.528982342,
    "conversions.delta": 29532.640850362,
    "conversions.pValue": 0.26076775078954,
    "conversions.isStatSig": 0,
    "conversions.reached": 735235,
    "conversions.reachedPercent": 0.49891935358004,
    "conversions.baseline": 715045.47101766,
    "conversions.lift": 0.028235307824002,
    "incrementalROAS": -0.1998564046528,
    "sales.test": 2112564.35,
    "sales.control": 528126.12,
    "sales.scaled": 2117249.607677,
    "sales.incremental": -4685.2576770443,
    "sales.delta": 113045.85347285,
    "sales.pValue": 0.94564373701777,
    "sales.isStateSig": 0,
    "sales.reached": 1103736.31,
    "sales.baseline": 1108421.567677,
    "sales.lift": -0.0042269636514411,
  }],
}

Result breakdowns

In addition to retrieving the results per objective, you may choose to breakdown the results by providing the breakdowns parameter.

curl -G \
-d 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_OBJECTIVE_ID>?fields=results&breakdowns=['cell_id']

The following are the available breakdown dimensions:

BreakdownValues

age

13-17, 18-24, 25-34, 35-44, 45-54, 55-54, 65+

cell_id

IDs of the available cells in the study.

gender

M or F

country

Two letter country codes (ISO 3166-1 alpha-2) e.g. US, GB, IN, AU. Currently supported only when queried in combination with cell_id i.e. like breakdowns=['cell_id','country']

The results will return multiple JOSN objects in the array based on the available breakdowns. For example, if cell_id is provided, the results will be broken down by the number of cells in the study. You may provide one or more breakdowns; however, the combination of breakdowns must at least 100 conversions from test or control groups in order for results to be displayed.

{
  "id": "<STUDY_OBJECTIVE_ID>",
  "results": [
  {
    "cell_id": "<CELL_ID1>",
    ...
    Default fields where the values are specific to the <CELL_ID1> breakdown
    ...
  },
  {
    "cell_id": "<CELL_ID2>",
    ...
    Default fields where the values are specific to the <CELL_ID2> breakdown
    ...
  }],
}