Lift Study

Set Up Studies

Set up a study with one or more groups, called cells. When you set up your study, Facebook randomizes the audience for your ads and assigns people to either the test or control group. After you run a study, Facebook calculates the difference between the test groups and control groups so that you evaluate the impact of your Facebook ads towards business goals.

To set up a study, make a POST call:

https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

You can set up a study with a single test group to see how Facebook ads lead to additional business. You can also set up a study with multiple test groups, which lets you determine what advertising approach works best for your audience.

Example - Set up a lift study with one test group

curl \
  -F 'name="new study"' \
  -F 'description="description of my study"' \
  -F 'start_time=1435622400' \
  -F 'end_time=1436918400' \
  -F 'cooldown_start_time=1433116800' \
  -F 'observation_end_time=1438300800' \
  -F 'viewers=[<USER_ID1>, <USER_ID2>]' \
  -F 'type=LIFT' \
  -F 'cells=[{name:"test group",description:"description of my test group",treatment_percentage:90,control_percentage:10,adaccounts:[<ACCOUNT_ID1>,<ACCOUNT_ID2>]}]' \
  -F 'objectives=[{name:"new objective",is_primary:true,type:"MAI",applications:[{id:<APP_ID>}]}]' \
  -F 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

To create a new study, provide the following:

Create a Test Group

To begin, determine how many people recieve your ads and how many people do not. You must create a test group when you set up the study; pass a list of JSON objects in cells under ad_studies. See Ad Study Cell, Reference. A test group contains the following information.

Example - Read test groups in a study

curl -G \
  -d 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<STUDY_ID>/cells

Example - Update or modify cell information and treatment and control percentages by providing the cell ID in cells

curl \
  -F 'cells=[{id:<CELL_ID>,treatment_percentage:80,control_percentage:20}]' \
  -F 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<STUDY_ID>

Example - Read all the studies that you created at ad_studies for your business

curl -G \
  -d 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

Set Up Multiple Test Groups

Set up a study with multiple test groups of Facebook users. This helps measure incremental impact of different Facebook strategies on business goals, such as using different ads targeting options. To set up a study with multiple test groups, provide a list of test groups in cells.

curl \
  -F 'name="new study"' \
  -F 'description="description of my study"' \
  -F 'start_time=1435622400' \
  -F 'end_time=1436918400' \
  -F 'cooldown_start_time=1433116800' \
  -F 'observation_end_time=1438300800' \
  -F 'viewers=[<USER_ID1>, <USER_ID2>]' \
  -F 'type=LIFT' \
  -F 'cells=[{name:"group A",description:"description of group A",treatment_percentage:50,control_percentage:20,campaigns:[<CAMPAIGN_ID1>]},{name:"group B",description:"description of group B",treatment_percentage:20,control_percentage:10,campaigns:[<CAMPAIGN_ID2>]}]' \
  -F 'objectives=[{name:"new objective",is_primary:true,type:"MAI",applications:[{id:<APP_ID>}]}]' \
  -F 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

control_percentage determines the holdout for each test group respective to the total population. For example, you have a study with two test groups: group A is 50% treatment with 20% control and group B is 20% treament with 10% control. This results in ~28.6%, or 20%/70% of the population in group A, to be control users and ~33.3%, or 10%/30% of the population in group B, to be control users.

The sum of treatment and control percentages across test groups normally should equal 100. However, it can be less than 100 for some specific use cases. For example, when you have three test groups that are split evenly at 33%.

You can update, add, and remove test groups in a study.

• To update an existing test group, refer to its ID in test group.
• To add a new test group, provide a new test group object.
• To remove a test group, simply omit it from cells when you update the study:

curl \
  -F 'cells=[{id:<CELL_ID1>,treatment_percentage:60,control_percentage:10},{name:"group C",description:"replacing group B",treatment_percentage:25,control_percentage:5,campaigns:[<CAMPAIGN_ID3>]}]' \
  -F 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<STUDY_ID>

Define Advertising Objectives

Define advertising objectives you want to measure and how you pass conversion data to Facebook. A lift study requires at least one objective. You cannot modify objectives after the study starts running. See Ad Study Objective, Reference.

Example - Create and add the MAI objective to a study

curl \
  -F 'name="new study"' \
  -F 'description="description of my study"' \
  -F 'start_time=1435622400' \
  -F 'end_time=1436918400' \
  -F 'cooldown_start_time=1433116800' \
  -F 'observation_end_time=1438300800' \
  -F 'viewers=[<USER_ID1>, <USER_ID2>]' \
  -F 'type=LIFT' \
  -F 'cells=[{name:"test group",description:"description of my test group",treatment_percentage:90,control_percentage:10,adaccounts:[<ACCOUNT_ID1>,<ACCOUNT_ID2>]}]' \
  -F 'objectives=[{name:"new objective",is_primary:true,type:"MAI",applications:[{id:<APP_ID>}]}]' \
  -F 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

If you use SALES and NONSALES and use Facebook pixel or Mobile App as event sources, you must provide a list of the event names that you want to capture for the objective. Facebook can then report results based on these specific conversion events.

Create an Objective

Create an objective by passing a list of JSON objects objectives when you create a new study. Objectives contain the following information:

You can also have multiple objectives per study. The result will be aggregated based on objectives. Below is an example of a study with multiple objectives—the MAI objective as seen previously and another SALES objective with Facebook pixel and application as measurement sources.

curl \
  -F 'name="another study"' \
  -F 'description="description of another study"' \
  -F 'start_time=1435622400' \
  -F 'end_time=1436918400' \
  -F 'cooldown_start_time=1433116800' \
  -F 'observation_end_time=1438300800' \
  -F 'viewers=[<USER_ID1>, <USER_ID2>]' \
  -F 'type=LIFT' \
  -F 'cells=[{name:"test group",description:"description of my test group",treatment_percentage:90,control_percentage:10,adaccounts:[<ACCOUNT_ID1>,<ACCOUNT_ID2>]}]' \
  -F 'objectives=[{name:"MAI objective",is_primary:true,type:"MAI",applications:[{id:<APP_ID1>},{id:<APP_ID2>}]},{name:"SALES objective",type:"SALES",applications:[{id:<APP_ID3>,event_names:["fb_mobile_purchase"]}],adspixels:[{id:<FB_PIXEL_ID>,event_names:["fb_pixel_purchase","fb_pixel_lead"]}]}]' \
  -F 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<BUSINESS_ID>/ad_studies

You can update, add, and remove objectives in a study by doing so at the study level similar to modifying test groups. To update an existing objective, refer to its ID in the objectives object. To add a new objective, provide a new objective object. To remove an objective, simply omit it from the objectives parameter when you update it.

Example - Update an objective's applications measurement sources and remove its adspixels measurement sources

curl \
  -F 'objectives=[{id:<OBJECTIVE_ID>,name:"new objective name",applications:[{id:<APP_ID>}],adspixels:[]}]' \
  -F 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<STUDY_ID>

Example - Read objectives for a study

curl -G \
  -d 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<STUDY_ID>/objectives

Reporting

Retrieve Objectives

A study's objectives are defined during the study setup. See the setup guide on how to define your study's objectives

You can read the objectives that were created for a study by making a GET call to the study's objectives edge.

curl -G \
  -d 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<STUDY_ID>/objectives

Retrieve Results

To retrieve results for an objective, you can make a GET call to the objective node by specifying results in the fields parameter. last_updated_results field also tells you when the results data was last updated.

curl -G \
  -d 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<STUDY_OBJECTIVE_ID>?  fields=results,last_updated_results

The resulting data is a JSON object, containing strings with the following default fields:

Sample response shown as parsed JSON for ease of reading:

{
    "last_updated_results": "2019-04-30",
    "id": "<STUDY_OBJECTIVE_ID>",
      "results": [{
        "population.test": 8008,
        "population.control": 8069,
        "population.reached": 7329,
        "impressions": 151928,
        "spend": 2367.74,
        "frequency": 20.72970391595,
        "buyers.test": 795,
        "buyers.control": 685,
        "buyers.scaled": 679.82153922419,
        "buyers.incremental": 115.17846077581,
        "buyers.reached": 720,
        "buyers.reachedPercent": 0.90566037735849,
        "buyers.baseline": 604.82153922419,
        "buyers.lift": 0.19043379460915,
        "buyers.delta": 60.074001807654,
        "buyers.pValue": 0.0016110673739798,
        "buyers.isStatSig": 1,
        "buyers.singleCellBayesianConfidence": 0.98727038188854,
        "buyers.multiCellBayesianConfidence": 0.78486645400638,
        "buyers.bayesianCILower": 30.473292849176,
        "buyers.bayesianCIUpper": 199.95922666997,
        "conversions.test": 871,
        "conversions.control": 736,
        "conversions.scaled": 730.43598958979,
        "conversions.incremental": 140.56401041021,
        "conversions.reached": 789,
        "conversions.reachedPercent": 0.90585533869116,
        "conversions.baseline": 648.43598958979,
        "conversions.lift": 0.21677391857774,
        "conversions.delta": 67.978575551168,
        "conversions.pValue": 0.00067032207485984,
        "conversions.isStatSig": 1,
        "conversions.singleCellBayesianConfidence": 0.99176024719258,
        "conversions.multiCellBayesianConfidence": 0.80160595182145,
        "conversions.bayesianCILower": 44.564010410212,
        "conversions.bayesianCIUpper": 236.54889081671,
        "advancedConversions.test": 870,
        "advancedConversions.control": 771.72138217184,
        "advancedConversions.scaled": 765.88732537268,
        "advancedConversions.incremental": 104.11267462732,
        "advancedConversions.baseline": 684.88732537268,
        "advancedConversions.lift": 0.1520143106323,
        "advancedConversions.informativeSingleCellBayesianConfidence": 1,
        "advancedConversions.informativeMultiCellBayesianConfidence": 0.819,
        "advancedConversions.bayesianCILower": 56.7105227,
        "advancedConversions.bayesianCIUpper": 156.976729,
        "incrementalROAS": 1.3104231791133,
        "sales.test": 54663.94,
        "sales.control": 51953.96,
        "sales.scaled": 51561.198621886,
        "sales.incremental": 3102.7413781138,
        "sales.reached": 49594.66,
        "sales.baseline": 46491.918621886,
        "sales.lift": 0.06673721950148,
        "sales.delta": 5708.9583320282,
        "sales.pValue": 0.37130299866185,
        "sales.isStatSig": 0,
        "sales.singleCellBayesianConfidence": 0.74013779586612,
        "sales.multiCellBayesianConfidence": 0.70927872163835,
        "sales.bayesianCILower": -4827.3888400561,
        "sales.bayesianCIUpper": 11330.643600358,
        "advancedIncrementalROAS": 1.4777176303831,
        "advancedSales.test": 54619.86,
        "advancedSales.control": 51510.417142093,
        "advancedSales.scaled": 51121.008857837,
        "advancedSales.incremental": 3498.8511421632,
        "advancedSales.baseline": 46095.808857837,
        "advancedSales.lift": 0.075903888636687,
        "advancedSales.informativeSingleCellBayesianConfidence": 0.97125,
        "advancedSales.informativeMultiCellBayesianConfidence": 0.7505,
        "advancedSales.bayesianCILower": 596.181983164,
        "advancedSales.bayesianCIUpper": 6470.00248918
    }]
}],
}

Breakdown Results

In addition to retrieving the results per objective, you may choose to breakdown the results by providing the breakdowns parameter.

curl -G \
  -d 'access_token=<ACCESS_TOKEN>' \
  https://graph.facebook.com/<API_VERSION>/<STUDY_OBJECTIVE_ID>?fields=results&breakdowns=['cell_id']

The following are the available breakdown dimensions:

The results return multiple JSON objects in the array based on the available breakdowns. For example, if cell_id is provided, the results are broken down by the number of cells in the study. You may provide one or more breakdowns; however, the combination of breakdowns must at least 100 conversions from test and control groups combined for results to display.

{
  "id": "<STUDY_OBJECTIVE_ID>",
  "results": [
  {
    "cell_id": "<CELL_ID1>",
    ...
    Default fields where the values are specific to the <CELL_ID1> breakdown
    ...
  },
  {
    "cell_id": "<CELL_ID2>",
    ...
    Default fields where the values are specific to the <CELL_ID2> breakdown
    ...
  }],
}