Lift Study

Create and run an experiment to measure your Facebook campaign's efficiency. Determine what ads strategy drives the most business impact. See Ad Study, Reference.

When you create a lift study, you create a randomized test group of people that see your ads and control group who don't see your ads.

You can securely share conversion data from your ad campaign with Facebook using Facebook Pixels, or App Events. Facebook determines the increased conversions generated from your campaign. We compare the number of conversions, people converting, and available sales revenue between test and control groups. Results appear in Ads Manager.

Setting Up Studies

Set up a study with one or more groups, called cells. When you set up your study, Facebook randomizes the audience the audience for your ads and assigns people to either the test or control group. After you run a study, Facebook calculates the difference between the test groups and control groups so that you evaluate the impact of your Facebook ads towards business goals.

To set up a study, make a POST call:

https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

You can set up a study with a single test group to see how Facebook ads lead to additional business. You can also set up a study with multiple test groups, which lets you determine what *advertising approach** works best for your audience.

Here is an example of how to set up a lift study with one test group.

curl \
-F 'name="new study"' \
-F 'description="description of my study"' \
-F 'start_time=1435622400' \
-F 'end_time=1436918400' \
-F 'cooldown_start_time=1433116800' \
-F 'observation_end_time=1438300800' \
-F 'viewers=[<USER_ID1>, <USER_ID2>]' \
-F 'type=LIFT' \
-F 'cells=[{name:"test group",description:"description of my test group",treatment_percentage:90,control_percentage:10,adaccounts:[<ACCOUNT_ID1>,<ACCOUNT_ID2>]}]' \
-F 'objectives=[{name:"new objective",is_primary:true,type:"MAI",applications:[{id:<APP_ID>}]}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

To create a new study, provide the following:

  • name of study.
  • description, or the study's purpose.
  • cooldown_start_time. Start of your pre-campaign quiet period. During this time, Facebook pauses all ads in the study to establish baseline data. We resume running ads once your study period starts. You do not need to establish a baseline for your study, you can set this value to equal start_time.
  • start_time of campaign active period, when ads start running. Study start time must be in the future.
  • end_time of campaign active period, when ads stop running.
  • observation_end_time. End of your post-campaign quiet period which starts when the study period ends. We attribute conversions during this period to your study. Facebook also pauses any ads in the study during this period. If you don't need this quiet period for your study, set equal to end_time.
  • cells in study that define test and control groups.
  • objectives of the study. See Defining Study Objective.
  • viewers. Share this study to a list of Facebook User IDs.
  • type of study. For Conversion Lift, the type should be LIFT.

Once the study starts, you cannot update start_time and treatment_percentage of the cells. You also cannot remove the associated objects, such as adaccounts or campaigns, of the test groups. You can still update the end_time and observation_end_time to a future time if the study has not yet ended and add new associated objects to test groups.

To run Reach and Frequency in conjunction with Lift measurement, you must set up a Lift study first and make sure the duration of the Reach and Frequency is within the duration of the Lift study.

Test Group

Determine how many people recieve your ads and how many people do not. You must create a test group when you set up the study; pass a list of JSON objects in cells under ad_studies. See Ad Study Cell, Reference. A test group contains this information:

  • name of the test group.
  • description of the test group.
  • treatment_percentage defines people who recieve your ads.
  • control_percentage defines a holdout percentage of the people who will not see ads. Treatment plus control percentages must equal 100.
  • List of ad entities, such as adaccounts or campaigns, that you want to study. Facebook runs and measures all ads under active ad entities during the study period.

To read test groups in a study:

curl -G \
-d 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_ID>/cells

You can update or modify cell information as well as treatment and control percentages by providing the cell ID in cells:

curl \
-F 'cells=[{id:<CELL_ID>,treatment_percentage:80,control_percentage:20}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_ID>

You can also read all the studies that you created at ad_studies for your business.

curl -G \
-d 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

Multiple Test Groups

Set up a study with multiple test groups of Facebook users. This helps measure incremental impact of different Facebook strategies on business goals, such as using different ads targeting options. To set up a study with multiple test groups, provide a list of test groups in cells.

curl \
-F 'name="new study"' \
-F 'description="description of my study"' \
-F 'start_time=1435622400' \
-F 'end_time=1436918400' \
-F 'cooldown_start_time=1433116800' \
-F 'observation_end_time=1438300800' \
-F 'viewers=[<USER_ID1>, <USER_ID2>]' \
-F 'type=LIFT' \
-F 'cells=[{name:"group A",description:"description of group A",treatment_percentage:50,control_percentage:20,campaigns:[<CAMPAIGN_ID1>]},{name:"group B",description:"description of group B",treatment_percentage:20,control_percentage:10,campaigns:[<CAMPAIGN_ID2>]}]' \
-F 'objectives=[{name:"new objective",is_primary:true,type:"MAI",applications:[{id:<APP_ID>}]}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

control_percentage determines the holdout for each test group respective to the total population. For example, you have a study with two test groups: group A is 50% treatment with 20% control and group B is 20% treament with 10% control. This results in ~28.6%, or 20%/70% of the population in group A, to be control users and ~33.3%, or 10%/30% of the population in group B, to be control users.

The sum of treatment and control percentages across test groups normally should equal 100. However, it can be less than 100 for some specific use cases. For example, when you have three test groups that are split evenly at 33%.

You can update, add, and remove test groups in a study. To update an existing test group, refer to its ID in test group. To add a new test group, provide a new test group object. To remove a test group, simply omit it from cells when you update the study:

curl \
-F 'cells=[{id:<CELL_ID1>,treatment_percentage:60,control_percentage:10},{name:"group C",description:"replacing group B",treatment_percentage:25,control_percentage:5,campaigns:[<CAMPAIGN_ID3>]}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_ID>

Defining Objectives

Define advertising objectives you want to measure and how you pass conversion data to Facebook. A lift study requires at least one objective. You cannot modify objectives after the study starts running. See Ad Study Objective, Reference. For example, create and add the MAI objective to a study:

curl \
-F 'name="new study"' \
-F 'description="description of my study"' \
-F 'start_time=1435622400' \
-F 'end_time=1436918400' \
-F 'cooldown_start_time=1433116800' \
-F 'observation_end_time=1438300800' \
-F 'viewers=[<USER_ID1>, <USER_ID2>]' \
-F 'type=LIFT' \
-F 'cells=[{name:"test group",description:"description of my test group",treatment_percentage:90,control_percentage:10,adaccounts:[<ACCOUNT_ID1>,<ACCOUNT_ID2>]}]' \
-F 'objectives=[{name:"new objective",is_primary:true,type:"MAI",applications:[{id:<APP_ID>}]}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies
NameDescriptionMeasurement Sources

SALES

Measure the lift in purchases. The reporting will contain dollar amount.

Conversion Pixels, Facebook Pixels, Mobile App, Offline Event Sets, and Custom Conversions.

NONSALES

Measures the lift in non-purchase conversions.

Conversion Pixels, Facebook Pixels, and Mobile App.

MAI

Measure the lift in mobile application installs.

Mobile App

If you use SALES and NONSALES and use Facebook Pixel or Mobile App as event sources, you must provide a list of the event names that you want to capture for the objective. Facebook can then report results based on these specific conversion events.

Measurement SourceEvent Names

Facebook Pixel

fb_pixel_view_content, fb_pixel_search, fb_pixel_add_to_cart, fb_pixel_add_to_wishlist, fb_pixel_initiate_checkout, fb_pixel_add_payment_info, fb_pixel_purchase, fb_pixel_lead, fb_pixel_complete_registration, custom

Mobile App

fb_mobile_activate_app, fb_mobile_complete_registration, fb_mobile_content_view, fb_mobile_search, fb_mobile_rate, fb_mobile_tutorial_completion, fb_mobile_add_to_cart, fb_mobile_add_to_wishlist, fb_mobile_initiated_checkout, fb_mobile_add_payment_info, fb_mobile_purchase, fb_mobile_level_achieved, fb_mobile_achievement_unlocked, fb_mobile_spent_credits

Create an objective by passing a list of JSON objects objectives when you create a new study. Objectives contain the following information:

  • name of the objective.
  • is_primary, a boolean spcifying if this is your primary advertising objective. A study can only have one primary objective.
  • type of objective: SALES, NONSALES, or MAI.
  • adspixels, list of Facebook Pixel IDs along with the relevant list of event_names per id if applicable.
  • applications, list of your mobile apps including relevant event_names per id.
  • offsitepixels, list of Conversion Pixel IDs if applicable.
  • offline_conversion_data_sets, list of Offline Event Set IDs if applicable. Currently, we do not support event breakdowns for Offline Conversion.
  • customconversions, list of Custom Conversion IDs if applicable.

You can also have multiple objectives per study. The result will be aggregated based on objectives. Here is an example of a study with multiple objectives—the MAI objective as seen previously and another SALES objective with Facebook Pixel and Application as measurement sources.

curl \
-F 'name="another study"' \
-F 'description="description of another study"' \
-F 'start_time=1435622400' \
-F 'end_time=1436918400' \
-F 'cooldown_start_time=1433116800' \
-F 'observation_end_time=1438300800' \
-F 'viewers=[<USER_ID1>, <USER_ID2>]' \
-F 'type=LIFT' \
-F 'cells=[{name:"test group",description:"description of my test group",treatment_percentage:90,control_percentage:10,adaccounts:[<ACCOUNT_ID1>,<ACCOUNT_ID2>]}]' \
-F 'objectives=[{name:"MAI objective",is_primary:true,type:"MAI",applications:[{id:<APP_ID1>},{id:<APP_ID2>}]},{name:"SALES objective",type:"SALES",applications:[{id:<APP_ID3>,event_names:["fb_mobile_purchase"]}],adspixels:[{id:<FB_PIXEL_ID>,event_names:["fb_pixel_purchase","fb_pixel_lead"]}]}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

You can update, add, and remove objectives in a study by doing so at the study level similar to modifying test groups. To update an existing objective, refer to its ID in the objectives object. To add a new objective, provide a new objective object. To remove an objective, simply omit it from the objectives parameter when you update it.

Here is an example of updating an objective's applications measurement sources and removing its adspixels measurement sources:

curl \
-F 'objectives=[{id:<OBJECTIVE_ID>,name:"new objective name",applications:[{id:<APP_ID>}],adspixels:[]}]' \
-F 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_ID>

To read objectives for a study:

curl -G \
-d 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_ID>/objectives

Reporting

Retrieving objectives

A study's objectives are defined during the study setup. See the setup guide on how to define your study's objectives

You can read the objectives that were created for a study by making a GET call to the study's objectives edge.

curl -G \
-d 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_ID>/objectives

For more details on objectives, refer to the Ad Study Objective reference documentation

Retrieving results

To retrieve results for an objective, you can make a GET call to the objective node by specifying results in the fields parameter. last_updated_results field will also tell you when the results data was last updated.

curl -G \
-d 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_OBJECTIVE_ID>?fields=results,last_updated_results

The resulting data is a stringified JSON object with the following default fields:

NameDescription

population.test

The number of people included in the test group of the experiment.

population.control

The number of people included in the control group of the experiment.

polutation.reached

The number of people in the test group that were reached by 1 or more ads associated with the lift study.

impressions

Number of billable ad impressions for ads associated with the lift study.

spend

The total amount spent up to the current point of your study.

frequency

The average number of times your ad was shown to each person in population.reached.

buyers.test

The number of buyers from the test group of the experiment.

buyers.control

The number of buyers from the control group of the experiment.

buyers.scaled

The number of buyers in the control group, factored-up by the ratio of population.test/population.control to produce a fair comparison for buyers.test.

buyers.incremental

The increased number of buyers as a result of running campaigns. Calculated as: buyers.test - buyers.scaled.

buyers.delta

The 90% confidence interval associated with the estimate for buyers.incremental.

buyers.pValue

The p-value associated with buyers.incremental.

buyers.isStatSig

1 = buyers.incremental is statistically significant at 90% confidence. 0 = buyers.incremental is not statistically significant at 90% confidence interval.

buyers.reached

The number of buyers reached by your ads.

buyers.reachedPercent

buyers.reached / buyers.test

buyers.baseline

buyers.reached - buyers.incremental

buyers.lift

buyers.incremental / buyers.baseline

conversions.test

The number of conversions from the test group of the experiment.

conversions.control

The number of conversions from the control group of the experiment.

conversions.scaled

The number of conversions in the control group, factored-up by the ratio of population.test/population.control to produce a fair comparison for conversions.test.

conversions.incremental

The increased number of conversions as a result of running campaigns. Calculated as: conversions.test - conversions.scaled.

conversions.delta

The 90% confidence interval associated with the estimate for conversions.incremental.

conversions.pValue

The p-value for conversions.incremental.

conversions.isStatSig

1 = conversions.incremental is statistically significant at 90% confidence. 0 = conversions.incremental is not statistically significant at 90% confidence interval.

conversions.reached

The number of conversions from buyers reached by your ads.

conversions.reachedPercent

conversions.reached / conversions.test

conversions.baseline

conversions.reached - conversions.incremental

conversions.lift

conversions.incremental / conversions.baseline

incrementalROAS

Incremental return on ad spend. Calculated as: sales.incremental / spend.

sales.test

The amount of sales from the test group of the experiment.

sales.control

The amount of sales from the control grooup of the experiment.

sales.scaled

The amount of sales in the control group, factored-up by the ratio of population.test/population.control to produce a fair comparison for sales.test.

sales.incremental

The increased amount of sales as a result of running campaigns. Calculated as: sales.test - sales.scaled.

sales.delta

The 90% confidence interval associated with the estimate for sales.incremental

sales.pValue

The p-value associated with sales.incremental.

sales.isStatSig

1 = sales.incremental is statistically significant at 90% confidence. 0 = sales.incremental is not statistically significant at 90% confidence interval.

sales.reached

The amount of sales from buyers reached by your ads.

sales.baseline

sales.reached - sales.incremental

sales.lift

sales.incremental / sales.baseline

Sample response shown as parsed JSON for ease of reading:

{
  "id": "<STUDY_OBJECTIVE_ID>",
  "results": [{
    "population.test": 1349292,
    "population.control": 336567,
    "polutation.reached": 738219,
    "impressions": 3349184,
    "spend": 23443.12,
    "frequency": 4.5368434028385,
    "buyers.test": 212300,
    "buyers.control": 50823,
    "buyers.scaled": 203748.63642603,
    "buyers.incremental": 8551.3635739689,
    "buyers.delta": 1536.450929913,
    "buyers.pValue": 0,
    "buyers.isStatSig": 1,
    "buyers.reached": 111199,
    "buyers.reachedPercent": 0.52378238341969,
    "buyers.baseline": 102647.63642603,
    "buyers.lift": 0.08330794426164,
    "conversions.test": 1473655,
    "conversions.control": 362552,
    "conversions.scaled": 1453465.4710177,
    "conversions.incremental": 20189.528982342,
    "conversions.delta": 29532.640850362,
    "conversions.pValue": 0.26076775078954,
    "conversions.isStatSig": 0,
    "conversions.reached": 735235,
    "conversions.reachedPercent": 0.49891935358004,
    "conversions.baseline": 715045.47101766,
    "conversions.lift": 0.028235307824002,
    "incrementalROAS": -0.1998564046528,
    "sales.test": 2112564.35,
    "sales.control": 528126.12,
    "sales.scaled": 2117249.607677,
    "sales.incremental": -4685.2576770443,
    "sales.delta": 113045.85347285,
    "sales.pValue": 0.94564373701777,
    "sales.isStateSig": 0,
    "sales.reached": 1103736.31,
    "sales.baseline": 1108421.567677,
    "sales.lift": -0.0042269636514411,
  }],
}

Result breakdowns

In addition to retrieving the results per objective, you may choose to breakdown the results by providing the breakdowns parameter.

curl -G \
-d 'access_token=<ACCESS_TOKEN>' \
https://graph.facebook.com/<API_VERSION>/<STUDY_OBJECTIVE_ID>?fields=results&breakdowns=['cell_id']

The following are the available breakdown dimensions:

BreakdownValues

age

13-17, 18-24, 25-34, 35-44, 45-54, 55-54, 65+

cell_id

IDs of the available cells in the study.

gender

M or F

country

Two letter country codes (ISO 3166-1 alpha-2) e.g. US, GB, IN, AU. Currently supported only when queried in combination with cell_id i.e. like breakdowns=['cell_id','country']

The results will return multiple JOSN objects in the array based on the available breakdowns. For example, if cell_id is provided, the results will be broken down by the number of cells in the study. You may provide one or more breakdowns; however, the combination of breakdowns must at least 100 conversions from test and control groups combined in order for results to be displayed.

{
  "id": "<STUDY_OBJECTIVE_ID>",
  "results": [
  {
    "cell_id": "<CELL_ID1>",
    ...
    Default fields where the values are specific to the <CELL_ID1> breakdown
    ...
  },
  {
    "cell_id": "<CELL_ID2>",
    ...
    Default fields where the values are specific to the <CELL_ID2> breakdown
    ...
  }],
}