Working asynchronously in Meta Content Library

Working asynchronously in Meta Content Library API enables researchers to efficiently manage and execute complex data queries without the need for immediate results. By organizing work into dedicated collections and leveraging asynchronous queries and jobs, users can track and reproduce research findings over time, and share them with other researchers within their cleanroom environment.

Before calling methods on the client objects in the examples on this page, you must first run the following code:

client <- import("metacontentlibraryapi")$MetaContentLibraryAPIClient

client$set_default_version("v6.0")

from metacontentlibraryapi import MetaContentLibraryAPIClient as client

client.set_default_version("v6.0")

Working with queries in the MCL API

Creating and getting queries and jobs

You create queries using the POST <path>/job endpoint. When a query is created it creates both a query and a job. The query can be referenced for further executions, the job represents the initial execution of this query. The response contains IDs for the newly created query and job. You can use these IDs to track and rerun the query.

The following example shows how to create a snapshot job, with a name and description, that returns Instagram post data:

response <- client$post(
    path="instagram/posts/job",
    params=list(q="Fete de la musique", since="2025-06-20", until="2025-06-23", mode="SNAPSHOT", name="Query on fete de la musique", description="I would like to do some music research")
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="instagram/posts/job",
    params={"q": "Fete de la musique", "since": "2025-06-20", "until": "2025-06-23", "mode": "SNAPSHOT","name": "Query on fete de la musique", "description": "I would like to do some music research"}
)
display(response.json())

The following example shows the format of the response from submitting a new job:

{
  "id": "2025-09-02-moe-joe",
  "creation_time": 1761582797,
  "update_time": 1761582830,
  "status": "IN_PROGRESS",
  "mode": "SNAPSHOT",
  "query_id": "2025-09-02-tka-fka"
}
  

You can then work with the created query and job using APIs under the async endpoint.

The following example shows how to get your queries:

response <- client$get(
    path="async/queries"
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.get(
    path="async/queries",
)
display(response.json())

When you get a single query, you receive its metadata for the query and, for queries that you created, a list of the jobs that have run the query. You can get your queries, as well as the queries of other researchers in your cleanroom environment that have "visibility": "PUBLIC" set on them. The jobs for other users' queries are not listed.

The following example shows how to get a query:

response <- client$get(
    path="async/queries/<some-query-id>"
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.get(
    path="async/queries/<some-query-id>",
)
display(response.json())

The following example shows the format of the response, including metadata about jobs that have run the query.

{
  "id": "2025-09-02-tka-fka",
  "creation_time": 1756838071,
  "update_time": 1756838071,
  "ownership": "OWNED_BY_YOU",
  "visibility": "PRIVATE",
  "platform": "Instagram",
  "entity_type": "Posts",
  "endpoint": "/meta-content-library/instagram/posts/job",
  "params": "{\"q\":\"Fete de la musique\",\"since\":\"2025-06-20\",\"until\":\"2025-06-23\"}",
  "name": "Query on fete de la musique",
  "description": "I would like to do some music research",
  "collection_id": "2025-09-02-sjd-rdr",
  "estimated_query_budget_cost": {
    "entities": 2089
  },
  "jobs": [
    {
      "id": "2025-09-02-moe-joe",
      "creation_time": 1756838075,
      "update_time": 1756838119,
      "status": "IN_PROGRESS",
      "mode": "SNAPSHOT",
    }
  ]
}

You can view the metadata for jobs on your queries. To see metadata for other researchers' queries, you must first copy the query. See Reproducibility: queries, below.

The following example shows how to get a job's metadata:

response <- client$get(
    path="async/jobs/<your-job-id>"
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.get(
    path="async/jobs/<your-job-id>",
)
display(response.json())

The following example shows the format of returned metadata for a job:

{
  "id": "2025-09-02-iyx-ldr",
  "creation_time": 1756835742,
  "update_time": 1756837280,
  "status": "COMPLETE",
  "mode": "SNAPSHOT",
  "last_snapshot_refresh_time": 1756837196,
  "query_id": "2025-09-02-uqc-ccc"
}
    

The following example shows how to get all jobs for all queries for which you have access:

response <- client$get(
    path="async/jobs"
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.get(
    path="async/jobs",
)
display(response.json())

Running a query

You can run a query multiple times. Each time it is run, a new job is created. You can get the query metadata to list all the jobs of the query, including the new job.

The following example shows how to run an existing query in a new snapshot job.

response <- client$post(
    path="async/queries/<your-query-id>/job",
    body=list(mode="SNAPSHOT")
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/queries/<your-query-id>/job",
    body={"mode": "SNAPSHOT"}
)
display(response.json())

You can also update query metadata. The following example shows how to POST a change to query metadata:

response <- client$post(
    path="async/queries/<your-query-id>",
    body=list(name="Query about music", description="I would like to see some bands")
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/queries/<your-query-id>",
    body={"name": "Query about music", "description": "I would like to see some bands"}
)
display(response.json())

The following example shows how to delete a query and all its jobs:

response <- client$delete(
    path="async/queries/<your-query-id>"
)
response$status_code

response = client.delete(
    path="async/queries/<your-query-id>",
)
display(response.status_code)

The following example shows how to delete a job:

response <- client$delete(
    path="async/jobs/<your-job-id>"
)
response$status_code

response = client.delete(
    path="async/jobs/<your-job-id>",
)
display(response.status_code)

Reproducibility: queries

Setting the "visibility" attribute of a query controls who can view it. If you set it to PRIVATE, then only you can view it. If you set it to PUBLIC, then other researchers on your cleanroom environment can also view it. After you update the visibility of a collection, remember to update the visibility of any queries in a collection that you want to differ from the collection visibility.

The following example shows how to make a query public:

response <- client$post(
    path="async/queries/<your-query-id>",
    body=list(visibility="PUBLIC")
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/queries/<your-query-id>",
    body={"visibility": "PUBLIC"}
)
display(response.json())

You can copy your queries or the public queries of other researchers in your cleanroom environment. This will allow you to run jobs on the copied queries and view the metadata of those jobs.

The following example shows how to copy a query.

response <- client$post(
    path="async/queries/<other-query-id>/copy"
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/queries/<other-query-id>/copy",
)
display(response.json())

Working with jobs in MCL API

After a job is created, you can use methods on it to get its status and data, and you can write its data to a file.

The following example shows how to get a job's status and data (if the job's status is COMPLETE):

job <- client$get_async_job(job_id="<your-job-id>")

# Display the job status
job$get_status()

# Display the job data
data <- job$get_data()
data

job = client.get_async_job(job_id="<your-job-id>")

# Display the job status
display(job.get_status())

# Display the job data
data = job.get_data()
display(data)

Finally, you can also write the job's data to a file with its write_data_to_file method, as shown in the following example:

job <- client$get_async_job(job_id="<your-job-id>")

job$write_data_to_file(directory="<your-directory>", filename="<your-filename>")

job = client.get_async_job(job_id="<your-job-id>")

job.write_data_to_file(directory="<your-directory>", filename="<your-filename>")

Collections

You can arrange your work into collections, which act like folders that contain groups of queries. This allows you to manage complex projects with multiple related queries in a clear, organized manner. Each collection and query carries metadata such as a name, description and ID, making it easier to track and manage.

The descriptions of SRE procedures in this document assume you have expanded the right-hand bar in SRE and selected the Collections tab.

Setting a default collection

A default collection shows any queries that you create and run. To set a collection to default, click the ellipsis (...) menu item to the right of the collection name in Asynchronous collections, then click Set as default. You can manage where queries in your notebook appear by changing which collection is default.

Setting the default property to True in code causes new queries that you create to be shown within it. Only one collection can be default at a time. If you make a collection default and you already add a default collection, the previous collection will have its default property set to False.

The following example shows how to set the default property:

response <- client$post(
    path="async/collections/<your-collection-id>",
    body=list(default=TRUE)
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/collections/<your-collection-id>",
    body={"default": True}
)
display(response.json())

You can remove the default status of a collection by clicking the ellipsis menu, then clicking Remove default. Or, you can simply set another collection as default, and the default status will be removed from all collections except the one you just set. If no collection has default property set to True, then new queries will have no collection ID. In SRE, these queries will be visible in All Queries.

Changes to your collections may take as long as 10 seconds to be shown in the Collections tab. You can click the circular arrow icon to refresh the view to see your changes immediately.

Sharing a collection

To share a collection with other researchers working in SRE, click the ellipsis menu, then click Update. In the Update dialog, expand the Visibility dropdown, choose Public, and then click Update. Note the ID of the collection, which has the format YYYY-MM-DD-xxx-xxx, where x represents a letter. These IDs are visible in the screenshot below.

The people you want to share this collection with must pass this ID to the cloning API. See Cloning public queries and collections.

Working with collections

This section contains examples of how to work with asynchronous collections and queries using the Meta Content Library (MCL) API. The new async endpoint contains API elements related to collections, queries and jobs.

The following example shows how to create a new collection:

response <- client$post(
    path="async/collections",
    body=list(name="Research project one", description="Research about music")
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/collections",
    body={"name": "Research project one", "description": "Research about music"}
)
display(response.json())

The following example shows how to get your collections:

response <- client$get(
        path="async/collections",
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.get(
    path="async/collections",
)
display(response.json())

You can get metadata and queries for your collections and for public collections that belong to other users in your cleanroom environment.

The following code example shows how to get metadata, including queries, for one collection:

response <- client$get(
    path="async/collections/<some-collection-id>",
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.get(
    path="async/collections/<some-collection-id>",
)
display(response.json())

You can also delete a collection. Note that deleting a collection does not delete the queries inside it. They will appear under All Queries. You must manually delete each individual query.

The following example shows how to delete a collection:

response <- client$delete(
    path="async/collections/<your-collection-id>"
)
response$status_code

response = client.delete(
    path="async/collections/<your-collection-id>",
)
display(response.status_code)

Reproducibility: collections

You can set the visibility attribute of a collection to PUBLIC or PRIVATE. When you change the value of this attribute for a collection, the visibility value of every query in the collection is updated to match. You can change the visibility of queries individually, resulting in a mix of public and private queries under either a public or private collection.

The following code example shows how to update the visibility attribute of a collection:

response <- client$post(
    path="async/collections/<your-collection-id>",
    body=list(visibility="PUBLIC")
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/collections/<your-collection-id>",
    body={"visibility": "PUBLIC"}
)
display(response.json())

You can copy another researcher's collection if it has been made public. Copying a collection also copies all of the public queries that are in the collection.

The following example shows how to copy a collection:

response <- client$post(
    path="async/collections/<other-collection-id>/copy"
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/collections/<other-collection-id>/copy",
)
display(response.json())

You can move queries to a collection. You can also set the collection_id property to null to remove it from all collections, which will cause it to appear under All Queries.

The following example shows how to move a query into a collection:

response <- client$post(
    path="async/queries/<your-query-id>",
    body=list(collection_id="<your-collection-id>")
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/queries/<your-query-id>",
    body={"collection_id": "<your-collection-id>"}
)
display(response.json())

The following example shows how to remove a query from all collections:

response <- client$post(
    path="async/queries/<your-query-id>",
    body=list(collection_id=NULL)
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/queries/<your-query-id>",
    body={"collection_id": None}
)
display(response.json())

Queries and jobs

Queries are calls to specific endpoints with specific query parameters. Queries can be executed asynchronously repeatedly to monitor changes over time. This gives you the ability to associate each query with multiple jobs or result files, capturing the state of the data at different points in time, letting you build a time series of query results, tracking how data evolves over time without losing historical context.

Queries also have metadata associated with them, such as the name, description and visibility properties (PUBLIC/PRIVATE) as well as extra metadata such as query API endpoint, query parameters, platform, and entity type.

Each execution of a query is handled by a job, which represents an execution of the query at a specific point in time. Each asynchronous job has a status, which can be one of IN_PROGRESS, COMPLETE, or FAILED. It also has a result file after the job completes. Each query has a list of jobs that represent all the times that the query has been run.

Rather than creating a copy of a query to run at a later time, researchers can instead re-execute a query to fetch the latest data in a new job. This flexibility supports ongoing monitoring and analysis. When a user re-runs a query, more of the query budget is consumed.

Rerunning a query to get current data

To run a query to get current data, click the ellipsis menu to the right of the query, then click Run. In the Run dialog, expand the Mode dropdown menu and click Live. Note that the UI informs you that rerunning the query will count toward your query budget.

Snapshots and verification

Researchers can reproduce and verify past analyses by sharing snapshots that preserve the modified versions of the original query results at the time of execution, even as live data changes or is deleted.

Meta deletes query results and other non-query files stored on its servers on a regular basis. Live mode data is available for 30 days, at most. You can, however, put an asynchronous job into snapshot mode, which allows you to fetch a modified version of your original query results for up to one year. Snapshots are periodically refreshed at most every 30 days, redacting data that has been deleted, fallen below a visibility threshold, or been made private, among other criteria. Entities that become ineligible will be shown to the user with no data fields, only with their unique MCL ID field and an "is_invalid_id": true field.

The following example shows how the data for a redacted item is returned.

{"id": "123456789012345",
 "is_invalid_id": true}

When a snapshot mode asynchronous job is periodically refreshed, it returns the same entities that were returned in the first run. Although fields in the entities can be marked as having invalid IDs, as noted above, original statistics data, such as the numbers of likes, will reflect the values from the first job. If an entity has changed, such as if the text of a Facebook post has been edited since the query was first run, you will receive the updated value. Each entity that has updated fields will contain an "updated_fields" object consisting of index-value pairs that name the fields that were updated. The original values in the fields are no longer available

The following example shows how the data for updated fields is returned:

...
{
"id": "123456789012345",
"updated_fields": {"0": "post_owner.name",
                   "1": "post_owner.username"}
}

Rerunning a query as a snapshot

To rerun a query as a snapshot, click the ellipsis menu to the right of the query, then click Run. In the Run dialog, expand the Mode dropdown menu and click Snapshot. Note that the UI informs you that rerunning the query will count toward your query budget. There is a limit of 100 snapshots per user. However, you can delete unused snapshots to stay under the limit.

Reproducibility

Using MCL API, you can mark your collection or queries PUBLIC so that other researchers can copy them. When you make a collection public, all queries that it contains are also made public. To make individual queries private again, you must change their statuses one by one. (You can also make all queries in a collection private by marking the collection as PRIVATE.) New queries always have a visibility value of PRIVATE, even if they are created in a default collection with PUBLIC visibility.

Cloning a collection or query happens in two steps. First, the collection and/or the source code of the queries are copied into your query or collection collection. After the copying is complete, updated versions of the results are cloned into the target collection by running all the queries that were copied over. The target collection is either the copied collection if you cloned a collection, or the default collection if you copied a query. If there is no default collection and you are using SRE, the query goes into All Queries.

Copying a collection will copy all PUBLIC queries and jobs under the respective collection. Copying a PUBLIC query will copy only jobs with both a status of COMPLETE and a mode of SNAPSHOT for each query.

The copy operation replicates the structure of the collection/query. After a copy operation is completed, SRE refreshes the copied snapshot queries asynchronously, so they are “IN_PROGRESS” and not yet accessible. When the operation is complete, you will have the original query results with the updated values as discussed in Snapshots and verification. (Original statistics data, such as the numbers of likes, will be unchanged from the values returned by the first job.) This query operation will count toward your query budget. The estimated query budget impact is shown in the estimated_query_budget_cost field displayed on the collection or query.

Cloning public queries and collections

You use MCL API to copy queries and collections, using the ID for the entity to clone. (This is the ID that has the form YYYY-MM-DD-xxx-xxx.) Typically, your colleague or the lead researcher will share the ID with you as part of your workflow.

Note: Remember that cloning a collection or query will impact your query budget because the results are retrieved for you after the collections and queries are copied over.

The following code example shows how to clone a collection:

response <- client$post(
    path="async/collections/<other-collection-id>/copy"
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/collections/<other-collection-id>/copy",
)
display(response.json())

The following code example shows how to clone a query:

response <- client$post(
    path="async/queries/<other-query-id>/copy"
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/queries/<other-query-id>/copy",
)
display(response.json())

Working with snapshots in the MCL API

Snapshots allow you to retain a potentially redacted version of the data that was returned when the query was first run. You can set the "mode” property to SNAPSHOT when you create a new job to cause its results to be redacted as user data is made private, deleted, or edited.

The following example shows how to create a new asynchronous snapshot query:

response <- client$post(
    path="instagram/posts/job",
    params=list(q="Fete de la musique", since="2025-06-20", until="2025-06-23", mode="SNAPSHOT", name="Query on fete de la musique", description="I would like to do some music research")
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="instagram/posts/job",
    params={"q": "Fete de la musique", "since": "2025-06-20", "until": "2025-06-23", "mode": "SNAPSHOT","name": "Query on fete de la musique", "description": "I would like to do some music research"}
)
display(response.json())

You can convert a LIVE job to a SNAPSHOT job. Converting a job to a snapshot will stop its data from being deleted at the first of the month.

The following example shows how to change a LIVE job to SNAPSHOT job:

response <- client$post(
    path="async/jobs/<your-job-id>/snapshot"
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.post(
    path="async/jobs/<your-job-id>/snapshot",
)
display(response.json())

When your snapshot displays a status of "Expired", you can fetch the metadata via the API to reload it. You will see a status of "Pending" while SRE runs the job and redacts the data.

The following example shows how to get metadata for a job:

response <- client$get(
    path="async/jobs/<your-job-id>"
)
jsonlite::fromJSON(response$text, flatten = TRUE)

response = client.get(
    path="async/jobs/<your-job-id>",
)
display(response.json())