Use segmentation to create an effect that separates someone from their background. You can then transform the environment behind them using textures and objects.

Segmentation can only identify people from the chest upwards, in the immediate foreground.

Using Segmentation

To create an effect with segmentation and a 2D background, you'll:

  • Make a texture called a segmentation mask texture.
  • Create a background using a rectangle, canvas and material.
  • Apply the segmentation mask texture to the material - and adjust some of the material's properties.

Creating a Segmentation Mask Texture

To start, select the Camera in the Scene tab.

In the Inspector panel, click + next to Segmentation.

You'll now see a Segmentation Mask Texture in the Assets panel.

Making your background

Next, insert a Rectangle - it'll automatically be inserted as a child of a Canvas.

You'll see a small rectangle in the middle of the scene. For a segmentation effect, we want the rectangle to fill the screen and automatically rescale to fit whichever device is using this effect. To do this, select Fill Parent in the Inspector panel.

The rectangle should now cover your scene.

Next create a material. At the bottom of the Inspector panel, click + next to Material, and select Create New Material.

Adding the Segmentation Mask

Select the material in the Assets panel. In the Inspector panel:

  1. Check the box next to Alpha. We're choosing Alpha because we want the mask to create a transparent shape.
  2. Apply the Segmentation Mask under Alpha.
  3. Check the box next to Invert, because we want to replace the background with our texture, not the person.
  4. Add whatever texture or color you want under Diffuse.

You'll now have a complete effect using segmentation!

Using Segmentation with 3D objects

You'll need to use 3 different layers to make an effect with segmentation and 3D objects in the background:

  1. The first layer is where you'll segment someone from their background.
  2. The second with contain your 3D objects.
  3. The third will be the background.

To start:

  1. Select Camera in the Scene tab.
  2. Click the + sign next to both Segmentation and Camera Texture in the Inspector panel.

Layer 1

Follow all the same steps as you would when creating a 2D segmentation effect.

This time, instead of selecting Invert under Alpha in the Inspector panel, apply the CameraTexture you created under Diffuse.

In the Layers tab, rename this layer Segmentation.

Layer 2

Create another rectangle, and set it to Fill Parent in the Inspector Panel. Create a Material for this rectangle, and choose a color or texture for this the material.

Apply this to a new layer, and label the layer Background.

Next to Render Mode select Transparent, so the 3D objects are visible through the background.

Layer 3

Import your 3D object under the Canvas.

Create and apply this object to a new layer. It's worth naming the layer something like Objects, to keep track.

This layer should be in the middle of the list of layers, to make sure it renders correctly.

The Layers tab should look like this, with the segmentation layer first and the background last in the list:

You'll now have a segmentation effect with 3D objects in the background!