Sample Effect: Visual Shaders

Learn to create an effect with visual shader patches. In this tutorial, you'll take custom texture and use patches to transform it. You'll adding color and change its position, so it looks and moves across the face like a realistic beam of light:



In this guide, you'll learn about:

Download the sample content to follow along with this tutorial. If you open the unfinished project in the sample content folder, you'll see we've already imported the texture you'll need - it's listed in the Assets Panel as lightLeak_tex, and looks like a beam of light.

Adding the custom texture, camera texture and segmentation

In the finished effect, a beam of light shines across the user's face. It isn't visible in the background because we've separated the foreground and background using segmentation.

Materials in both the foreground and background also have the camera texture applied to them. In the foreground, we'll combine segmentation and the camera texture with the light leak texture, and a gradient effect too. In the background, we'll just add a gradient.

Adding segmentation

In Spark AR Studio, segmentation effects are created using a texture called a segmentation mask texture. Start by inserting the 2 rectangles that will render the textures in the scene.

  1. Click Add Object, and select a Rectangle. It'll automatically be inserted as a child of the canvas. The canvas ensures rectangles scale up and down with the size of the device screen.
  2. Right-click the canvas, and select a Rectangle.

To help you keep track, rename:

  1. The first rectangle in the list background.
  2. The second rectangle in the list user.

Your Scene Panel should look like this:



Next, create a material for each rectangle:

  1. Select each rectangle in the Scene Panel.
  2. Click + next to Material in the Inspector.


Like before, it's worth renaming the materials. Rename them user_material and background_material, to correspond with the rectangles they're applied to.

Finally, resize the rectangles so they fill the screen of the device:

  1. Select the rectangles - you can select them both at the same time by holding command on your keyboard if you're using a Mac, or control on a Windows machine.
  2. Click in the first box next to Size in then Inspector, and select Fill Width.
  3. Click in the second box next to Size, and select Fill Height.

Now create the texture you'll use to add segmentation - the segmentation mask texture:

  1. Select the Camera in the Scene Panel - it's listed underneath Device.
  2. In the Inspector, click + next to Segmentation.


You'll see the texture in the Assets Panel, listed as segmentationMaskTexture0.

To render segmentation in the scene, apply this texture to both the materials you've created:

  1. Select the user_mat in the Assets Panel.
  2. In the Inspector, tick the box next to Alpha.
  3. Click the dropdown next to Texture, and select Segmentation Mask Texture from the list.
  4. Follows these steps for background_mat, too.

Your project should look like this:

The camera texture

The camera texture is a video of what's being captured by the camera when the effect is open on a device. Applying the camera texture will show the user and background in the effect again.

To create the camera texture:

  1. Select the Camera in the Scene Panel.
  2. In the Inspector, click + next to Texture Extraction.

You'll see the texture listed in the Assets Panel, as cameraTexture0.

You'll finish creating this material in the Patch Editor, so you can combine multiple textures in the same material. Open the Patch Editor, by selecting View in the menu bar, then Show/Hide Patch Editor.

Next create a patch to represent the Camera Texture. You can do this by dragging it from the Assets Panel, into the Patch Editor.



You'll also need to create a patch to represent the Diffuse Texture property of each material. You can do this by:

  1. Selecting each material in the Assets Panel - you can select them both at the same time.
  2. Clicking the arrow next to Texture in the Inspector:


You'll see two new patches in the Patch Editor:



Next, connect the RGBA output in Camera Texture to the Diffuse Texture port in background_material.



If you change the video to your own camera, you'll see the background has appeared in the Simulator.

Adding the texture

Now it's time to complete user_material, by combining it the camera texture and custom texture, using an Add patch.

The texture is listed in the Assets Panel as lightLeak_tex. Drag it into the Patch Editor to create a patch.

To combine this texture with the camera texture, you'll use an Add patch. Right-click in the Patch Editor, and select an Add patch from the menu. You'll need to to change the data type of the Add patch to Vector 2 - because we're working with 2D textures:



Connect:

  1. The RGBA port in the CameraTexture and lightLeak_tex patches to the Add patch.
  2. The output in the Add patch to the Diffuse Texture port in user_material.

The graph should look like this:

You'll see the texture in the Simulator and Viewport:

Adding a color gradient

Now let's add the gradient effect. You'll need to break some of the connections between patches you've already made. It's simple to do this - just click on a cable connecting one patch to another, and press delete on your keyboard.

Disconnect:

  1. The connection between Camera Texture and the Diffuse Texture port in background_material.
  2. The connection between the Add patch and the Diffuse Texture port in user_material:


Now create the patches you'll need for the gradient effect. Right-click in the Patch Editor and select:

  1. A Gradient and Gradient Step patch.
  2. 2 Multiply patches.
  3. An Add patch.


You'll use:

  1. The Multiply patches to multiply the strength of the color gradient.
  2. The Add patch to combine the gradient with the camera texture and custom texture we've already added.

So, connect:

  1. The output of the purple Gradient patch to the input of the Gradient Step patch.
  2. The Output port in the Gradient Step patch to the input port in each Multiply patch.
  3. The output of one Multiply patch to the Diffuse Texture port in the background material patch:


To complete the gradient effect in the background of the scene, you'll need to make some edits to background_material. In the Inspector:

  1. Uncheck the checkbox next to Alpha.
  2. Under Render Options, change the Blend Mode to Add.
  3. Change the Opacity to 30%.

Next, make some adjustments to the Gradient Step patch:

  1. Change the color next to Previous Step to a pale yellow and the color next to New Color to white. You could pick different colors for a different effect.
  2. Set the Start Range to 0.6.

The shape of the gradient will be set to Horizontal by default. For this effect, change the shape of the gradient by clicking on the patch, and selecting Circular from the menu.

Then, connect the output of the Add patch connected to the two orange texture patches, to the bottom input in the Add patch you've just created:



Then, connect the output of the Add patch connected to the two texture patches, to the bottom input in the Add patch you've just created:



Now connect:

  1. The output of the second Add patch to the Diffuse Texture input in user_material.
  2. The output of the remaining Multiply patch to the input in the second Add patch.


The color effect is super bright:



To adjust this, change the values in the Multiply patches. Change:

  1. The value in the Multiply patch at the top of the graph to 0.7.
  2. The value in the Multiply patch at the bottom of the graph to 0.2.

We experimented to work out the best values for the Multiply patches. You could make the color strong or weaker by adjusting these numbers.

At this point, your project should look like this:



At the moment, the texture looks flat and unrealistic. To add realism, we'll make the light leak texture bend and distort relative to the camera texture, so it appears to fall in three dimensions across the face.

Distorting the Texture

In the finished effect, we've grouped the collection of patches that distort and transform the texture. It's called Texture Distortion Shader:



When it's connected to the rest of the graph, it combines and distorts the camera texture and custom texture, adding realism to the effect. You can take a look at the patches that make up the Texture Distortion Shader by clicking the icon in the right corner of the patch group, to expand it.

This part of graph works by using Vertex Attribute patches to read UV values, which define how a texture is laid across a surface. In this case, the surface is the rectangles our materials are applied to. This graph will take those values - normalized from 0 to 1 horizontally and vertically. It will then offset them using RG and RGB values from the camera texture - using a Multiply patch to amplify the effect.

These UVs, distorted by the camera texture and applied to the light leak texture, will causes the beam of light to distort relative to the camera texture, by looking up pixels in the light leak texture using the Texture Sampler patch. This gives the effect of the light rolling across a 3D face, even though the distortion is happening based on the RGB values, not actual depth or 3D information.

So you can quickly build this part of the effect, we've already added most of the patches you'll need for the Texture Distortion Shader and connected them in a graph. The graph is listed as an asset in the Assets Panel, under Patches. It's called Distortion Shader:



Drag and drop the Distortion Shader from the Assets panel into the Patch Editor. Right-click on the patch group, and select Ungroup. The graph will look like this:



We still need to adjust the values in these patches, and connect them to the textures and materials, so you're still going to learn exactly what each patch is doing.

You'll need to create a couple more patches too. Right-click in the Patch Editor and select:

  1. 2 Vertex Attribute patches.
  2. A Pack patch.

Change:

  1. The value in the Vertex Attribute patches to Texture Coordinates, to read the texture coordinates from the rectangles.
  2. The Type in the Pack patch to Vector 2.


Connect the output of one Vertex Attribute patch to the UV input in the Texture Sampler at the start of the graph:



Now connect the output of the other Vertex Attribute patch to input in the Add patch that's connected to the UV input in the second Texture Sampler:



Then connect the output of the Multiply patch at the end of the graph, to the input in the Pack patch:



Next we'll connect these patches to the rest of the graph.

Connecting the camera texture and custom texture

Like before, you'll need to break some connections again to do this. Disconnect the CameraTexture and LightLeak_tex patches from the rest of the graph.

You can delete the Add patch that was connected to lightLeak_tex - we don't need it anymore:



Instead, connect:

  1. The RGB port in lightLeak_tex to the Texture port in the Texture Transform patch.
  2. The RGB port in CameraTexture to both the Texture input in the Texture Sampler patch, and the top input in the final Multiply patch in the graph:

Connecting the materials

Go over to the right end of the graph, and connect the output port in the Pack patch to the remaining input port in the remaining Add patch. The end of the graph should look like this:



Now it's time to adjust the values in the patches.

Adjusting values

Start with the Pack patch at the end of the graph. Change the input value in the second row to 1:



You'll see the color in the scene, but the textures are really dark.



So you can see the texture, adjust the value in the Multiply patch at the bottom of the graph. Change it to -0.1. Then, change the value in the Swizzle patch to xy.



Let's adjust the brightness now. Do this by changing the input value in the Multiply patch that sits between the Texture Transform and Add patches to 1.5.

Change the value in the Add patch connected to it to 0.5. The Add patch lifts the black levels of the light leak texture, so when it'is multiplied over the camera texture, it doesn't darken it too much.



Next let's flip the custom texture over adjust the rotation and scale of the texture in the 2D Transform Pack:

  1. Next to Translation, change X to -0.09 and Y to 0.013.
  2. Next to Scale, change X to -1.1.
  3. Change the Rotation to 180.
  4. Next to Pivot change X to -0.5 and Y to 0.5.

And there we have it - you've used visual shaders to combine and distort the camera texture and a custom texture, creating the appearance of a beam of light:

Normalizing values

Now you have the texture in the scene, you're ready to add interaction - so the texture reacts to the user's face, and moves when the screen pans. You'll also add animation, to make the beam of wobble slightly from side to side.

This part of the effect will use the face tracker to detect the position of the face, the Screen Pan patch to detect screen interaction, and the Loop Animation patch to animate the texture.

The values these patches generate are high. If you connected, for example, the Screen Pan patch directly to the rest of the graph, the tiniest touch of the screen would make the texture move very dramatically.

To solve this problem, you'll normalize these values using a Divide patch. This will divide the values outputted by these patches by a high number, so they're much smaller.

You'll also create Add patches, which you'll use later to connect the animation patches and the Screen Pan patch to the rest of the graph.

Right-click in the Patch Editor and select 2 Add patches and a Divide patch.

Change the Type of the Add patches to Vector 2, because these patches will move the texture in 2D space - on the X and Y axes. Click on the patches, and change the Type to Vector 2.

Connect them, so your graph looks like this:



Then:

  1. Connect the output of the Divide patch to the Translation input in the 2D Transform Pack patch.
  2. Change the value in the Divide patch to 1000.

Adding the face tracker

The face tracker will detect the movement of the user's face. This data will be used to drive the movement of the beam of light.

First you'll need to add a face tracker to your project:

  1. Click Insert in the Menu bar.
  2. Select Face Tracker from the menu.

To create a patch representing the face tracker, drag and drop it into the Patch Editor.

You'll also need to select an Unpack, Pack and Multiply patch from the menu.

You'll use Unpack to split the X,Y and Z data stream representing the 3D Position of the face into individual channels. The Pack patch converts this data in to a Vector 2 data stream - X and Y. This data can then be read by the 2D Transform Pack patch.

You'll edit the Multiply patch to amplify the effect of the movement of the face, causing the light leak texture to move a bit faster in line with the face's movement.

Like with other patches in this effect, change the Type of the Pack patch to Vector 2, and the Multiply patch to Point 2D.

Connect the patches, so your graph looks like this:



Finally, change the bottom values in the Multiply patch to -5 and -7, to amplify the movement of the face.

Adding the animation

Next we'll add extra interest to the effect with an animation that makes the beam of light move slightly.

From the menu in the Patch Editor, select a:

  1. Loop Animation patch.
  2. Transition patch.

Change the Type of the Transition patch to Point 2D.

Connect:

  1. The Progress port in the Loop Animation patch to the Progress port in the Transition patch.
  2. The output in the Transition patch to the lower input in the Add patch.

The section of the graph should look like this:



To set up the animation, adjust the values in the Loop Animation and Transition patches:

In Loop Animation:

  1. Check the box next to Enable.
  2. Change Duration to 3.
  3. Check the box next to Mirrored.

In Transition, adjust the values to set where the position on the screen that the light beam will move between, and how it will move:

  1. Next to Start, set X to -50 and Y to -120.
  2. Next to End, set X to 50 and Y to -150.
  3. Change the Curve to Quadratic In-Out.

Here's how the patches should look:



The light beam should now move slightly across the face. Just one more step now - adding an interactive element to the effect using the Screen Pan patch.

Adding interactivity

Right-click in the Patch Editor, and select a Screen Pan patch. This patch will detect when a user moves their finger along the screen in one continuous motion.

Connect the 2D Offset port in Screen Pan to the remaining port in the second Add patch:



You've now completed the effect!

Learn more