There may be differences between your version of AR Studio and this tutorial because the product is currently in beta and we update it regularly.

Tutorial: Separating People from Backgrounds using Segmentation

Something Went Wrong
We're having trouble playing this video.

In this tutorial you'll learn how to separate the people in your scene from the background.

You'll do this by using segmentation. When you combine segmentation with a feature called texture extraction, you can create textures based on parts of the scene. Using these features you can replace the background behind the person.

Download the sample content to follow along. If you open the finished project, you'll see that you're going to build a simple background replacement effect.

Inserting Segmentation

To get started, open AR Studio and create a new project. First:

  1. Click Insert.
  2. Select Segmentation.
  3. Click Insert.

You should now see segmentation listed in the Scene tab. You can't see it in the Viewport, but segmentation has created an outline around the person in the scene.

Extracting Textures

Next, you'll need to create a texture using the contents of the outline. Make sure that segmentation is selected in the Scene tab, go to the Inspector panel and then click + next to Texture Extraction. Your new segmentation texture should now be listed in the Assets panel.

Once you've done that, select Camera in the Scene tab, go to Texture Extraction in the Inspector panel and click the +. You should see a camera texture listed in the Assets panel as well.

If you select the segmentation texture in the Assets panel, you'll see that it has 2 properties. You can use:

  1. Edge Softness to feather the outline of the person.
  2. Mask Size to adjust the size of the outline.

Inserting and Editing Rectangles

Now that you have the textures you need, you'll need to insert a surface to apply them to. So click Insert, select Rectangle and then click Insert.

You should see a canvas with a rectangle nested below it in the Scene tab and a 2D object in the scene.

For this effect, the rectangle needs to fill the screen. If you look at the Viewport, you'll see a large green rectangle. This indicates the size of the device screen, as does the Simulator. To get these dimensions, go to the Scene tab, click Device and note down the Screen Size values.

For me, that's 375 and 667, because I have the Aspect Ratio in the Toolbar set to iPhone 8 and the parent canvas takes the size of the selected device in screen space.

Now, go back to the rectangle's Inspector panel:

  1. Change the Size values to match those that you just noted down.
  2. Go to Flexibility and select each of the arrows - doing this means that whichever device you choose from the Aspect Ratio menu in future, the size of the rectangle will always fill the Simulator.

The rectangle should now be the only thing you can see in the Simulator.

Creating Materials

Next, you'll need to create the material to apply to your rectangle. To do this:

  1. Go to Material in the Inspector panel and click +.
  2. Double-click on the material you've selected to inspect its properties.
  3. Change Shader Type to Flat, to make the the rectangle doesn't react to the default lighting.

Once you've done that, go to Diffuse, click the arrow next to Texture and select the camera texture from the menu. This renders the camera feed onto the rectangle.

Then, go to Alpha, check the box, click the arrow next to Texture and select segmentation texture from the menu. This cuts away the rest of the rectangle and lets you mask the person in the scene.

Creating Backgrounds

Now you can add your custom background to the scene. The first thing you'll need is another rectangle, so click Insert, select Rectangle and click Insert.

Change the size of the rectangle to match the first one. The rectangle should now fill the Simulator. You'll also need to create the background material, so click the plus sign next to Material and select Create New Material from the menu. Double-click on the material you've created to inspect its properties.

We're going to use a basic color texture for this material, but you can use another texture if you'd prefer. To create the color texture, click in the box next to Color to open the color wheel and choose the color you want to use. We've chosen blue. The background should now be covering the Simulator.

Using Layers

To make the person visible and place the background behind the face, you'll need to use layers to adjust the render order. Go to the Layers tab and click the plus sign to create a new one. Move the new layer to the bottom of the tab, so that it is rendered first.

Then, go to the Scene tab, select the second rectangle, go to the Inspector panel and change the layer to the new one. You should now see the custom background behind the person in the scene.