There may be differences between your version of AR Studio and this tutorial because the product is currently in beta and we update it regularly.

Interacting With Objects in Your Surroundings

Something Went Wrong
We're having trouble playing this video.

In a previous tutorial, we explained how to place 3D objects in the world around you. While you could rotate your device towards and away from these 3D objects in your surroundings, they remained at a fixed distance from the camera and you couldn't move closer or further away from them.

In this tutorial, you'll learn how to add 3D objects that appear fixed in the world, rather than at a fixed distance from the camera, as well as interact with them and manipulate their position and scale.

Please download the sample content to follow along.

If you open the finished effect in the sample content folder, you'll see that the effect you're going to build is a mug of coffee that can be placed in your surroundings.

Inserting and Editing Plane Trackers

To get started, open AR Studio and click Create New Project. We want to make sure that the mug is placed on horizontal surfaces in the world, not just floating in space. To do this you'll need a plane tracker, which will detect an infinite, horizontal plane in your scene, so click Insert, select Plane Tracker and click Insert. You should now see a plane tracker listed at the bottom of the Scene tab. It won't be a child of the Camera node because it's tracking in world space, not camera space. You should also be able to see the plane tracker represented in the Viewport as a blue square with a cross inside. You may have noticed that your Viewport and Simulator look a little different now. The video has disappeared and the camera has switched from Front mode to Back, if it wasn't already. This is because the plane tracker detects planes in your surroundings and the back camera is typically used for capturing moments in your surroundings.

Please keep in mind that you can only have one plane tracker in a scene and, at the moment, it can only track horizontal planes. You won't be able to place the bottom of the mug onto a wall.

It's important that the plane tracker finds a plane as soon as the effect is opened, so make sure the plane tracker is selected in the Scene tab, go to the Inspector panel and make sure that Auto Start is checked.

Importing 3D Objects

Next, you'll need to import the mug and insert it as a child of the plane tracker. Right-click on the plane tracker in the Scene tab, select Insert, then select 3D Object and open mug.dae from the sample content folder on your computer. There should now be a 3D object listed in the Objects folder of the Assets panel and nested below the plane tracker. You should also see a model in the shape of a mug in your scene, although it doesn't really look like a mug just yet.

Importing Textures and Creating Materials

Now that the 3D object is in your scene, you can make it look more like a mug by applying a material to it. If you look at the mug in the Scene tab, you should see 3 meshes nested below it. Each one of these meshes will need a separate material, which you can create in AR Studio.

First, click the plus sign in the Assets panel, select Import Assets... from the menu and open the 2 textures files in the sample content folder on your computer. Next, go to mug.dae in the Scene tab, select the first child, also called mug, and go to the Inspector panel. Then, click the arrow next to No material selected and select Create New Material from the menu. The outside of the mug should have changed from checked to solid in the Viewport. Double-click on the material you've just created to open its properties.

Next, go to Diffuse, click the arrow next to Texture and select mug_color from the menu. The mug in your scene should have changed color, but it probably looks quite dark and flat at the moment. To make it look more realistic, check the box next to Specular. This property defines the level of shine and the highlight color of a surface. For this effect, we'll use a color. Click the arrows next to Type and change it to Color. We're going to choose a light beige color from the palette, but you can pick whatever color you like. Once you've done that, change the Smoothness property. That looks a little much, so we recommend you change it to 34%. Then, go to Render Options and check the box next to Double Sided.

The outside should now look more like a mug but, if you look at it, you'll see that it isn't empty. There's a surface for a material inside too. To create this material, select coffee in the Scene tab, click the arrow next to No material selected and select Create New Material from the menu. The inside of the mug should now have changed from checked to solid too. Double-click on the material you've just created to open its properties, then add mug_color - the same texture as before - as the texture below Diffuse. Then, check the box next to Emission. The mug should now look like it has coffee in it.

You'll notice that the mug still looks like it's sitting on a checked square. This is where the shadow of the mug will go. To do this, select shadow_plane in the Scene tab, create a new material and inspect its properties. Since this is supposed to be a shadow, we don't want the material you've created to reflect light. So, change Shader Type to Flat. Once you've done that, go to Diffuse, click the arrow next to Texture and select mug_shadow from the menu. To finish this material, go down to Render Options, change Blend Mode to Multiply and then change the Opacity. Hm, that doesn't look quite right. We recommend 77 or 78%. Once you've done that, check the box next to Double Sided. There should now be a small shadow around the mug in the Simulator, but the mug itself is still quite dark. We'll get to that next.

Inserting and Editing Lights

If you look in the Scene tab, you'll notice that your scene has some lights inserted by default: a directional light and an ambient light. These help the objects in your scene look more realistic. To illuminate this effect properly though, you'll need another directional light, so click Insert, select Directional Light and then click Insert. You should now see another directional light in the Scene tab and represented in the Viewport.

To make sure that all of the lights in your scene are the correct color and intensity, and pointed in the correct direction, you'll need to edit them. Let's start with the first directional light. First, selected it in the Scene tab and go to the Inspector panel. Then, click in the box next to Color and select one from the color palette. We're going to use a light beige again, but you can choose whatever color you like. Next, let's change the Intensity. That's a little dark, so let's move it into the 50s. Once you've done that, you'll need to adjust where the light focuses. You can do this with the Rotation transformations by changing the X value to -154, the Y value to 28 and the Z value to -40.

For the ambient light, go to Intensity and increase the percentage. Experiment with what feels right for you - we've found that 63% should do it. Then, select the second directional light, pick a new color - again, we've gone for the light beige - and change Intensity to the same value as the other one - 57% in our case. Then, go to Rotation and change the X value to -14, the Y value to -48. You can leave the Z value as 0.

Excluding Layers

To make sure that the 3D objects in your scene render in the correct order, you'll have to adjust the layers. 3D objects are all assigned to the same layer by default, but we can add more. To do this, go to the Layers tab and click the plus sign twice. You should now see 2 more layers in the tab.

Let's start by renaming the layers. This'll make it easier to refer to them later. Rename the top layer, coffee_layer, rename the second layer down, shadows_layer, and rename the bottom layer, mug_layer. Once you've done that, drag shadows_layer to the bottom of the list, then inspect its properties and change Render Mode to Transparent.

Now you'll need to re-assign the objects back in the Scene tab. First, select coffee, go to the Inspector panel, click the arrows next to the current layer and pick coffee_layer from the menu. Then, make sure that all 3 lights, the plane tracker and mug are assigned to mug_layer. Once you've done that, re-assign mug.dae and shadow_plane to the shadows layer.

The objects are on the correct layers, so now you'll need to exclude some of the layers from the effects of the directional lights. This means that the lights won't illuminate the objects in those layers. To do this, select the first directional light, go to the Inspector panel, click the plus sign next to Exclude and select coffee_layer from the menu. Then, select the second directional light and exclude the shadows layer. Your object should now render correctly.

Adding Interactivity

Your mug is now starting to take shape, but let's go one step further and add scripting, so that you can interact with it in your surroundings. To add a script to your project, go to the Assets panel, click the plus sign and select Create New Script. You should now find a file called script in the Scripts folder.

We've provided a script for you in the sample content folder, which you can open in your chosen script editor.

If you look at the code, you'll see that the first few lines enable the Scene, Reactive and TouchGestures modules. We're including the TouchGestures module because we want to use the taps, pinches and other gestures performed on a mobile device screen to trigger interactions. Lines 14 to 15 find the 3D object and the plane tracker in the scene. If you look at lines 18 to 20, you'll see the onTap class specified. This means that when the screen is tapped, the plane tracker will find a plane at that point and the mug will move to the position of that tap. The onPan class in lines 22 to 24 refers to the continuous movement of a finger on the screen. Doing this will make the plane tracker find plane after plane as the finger moves, so the mug will follow the finger across the screen. Lines 26 to 35 refer to the onPinch class. Pinching the screen will make the mug bigger and smaller, whereas onRotate on lines 37 to 40 will rotate the mug if the action is performed on the mug itself.

Now that you know what each part of the script will do, highlight it all and copy it. Then, go back to the Assets panel in AR Studio, open script, paste the script and save it.

Adding Capabilities

If you click Run in the toolbar now, you should see the Console appear and display an error. This is because your project doesn't currently support the touch gestures we discussed earlier. To add this support, go to the menu bar, click Project, then Edit Properties, then click the Capabilities tab. You should see Touch Gestures listed there already but, if you click the arrow, you'll see that none of the specific gestures have been added. Check all of the boxes, except for the last one, because these are all referenced in the script you added earlier.

Now, if you click Run you won't see an error in the Console.

Previewing on Mobile Devices

Your effect is nearly ready! Now's the time to preview and test it. The best way to do this is to mirror the effect from your computer to your mobile device, using the AR Studio Player app, so connect your mobile device to your computer with a USB cable, open the AR Studio Player app on your mobile device and click Mirror in the AR Studio toolbar. You may have to wait for a moment or two while the textures are compressed.

You should now see the mug in the world on your mobile device! Test the gestures by moving it around or changing its size.

Congratulations, you've built an effect that lets you interact with objects in your surroundings!


To recap, you've learned about using a plane tracker to place a 3D object on an infinite, horizontal plane. You've also learned how to write a script that lets you interact with and manipulate the object in the world.