Building an AR Feature in Android: From Zero to Working

The full pipeline for a tap-to-place augmented reality feature on Android — ARCore session, SceneView rendering on Filament, plane detection, object placement, and the production pitfalls I hit in the first week.

A furniture client in Delhi wanted "try it in your room" for a shelf unit — the one feature every furniture e-commerce site advertises and almost none actually ship. The brief looked simple: point the camera at the floor, the shelf appears, tap to place it. I had built computer-vision pipelines before, but never an AR feature in a shipping Android app. So I went from zero to working in one week, and the reality was a specific, learnable sequence of steps — not magic, and not a library you can blindly drop in.

This article is that sequence. If you have an AR feature on your roadmap, this gets you from an empty project to a shelf you can place on your office floor, with the failure modes listed so you do not have to hit them all yourself. We build in this order: prerequisites, dependencies and manifest, the ARCore session, the scene view, a 3D model, tap-to-place, occlusion and lighting, then error handling and testing.

Step 1: Check the Prerequisites First

Before a single line of code, the hardware reality: ARCore needs a physical device with the camera, IMU, and depth capabilities to track the world. Emulators are effectively useless for this. On a mid-range Android phone the feature works; on older devices or budget hardware without ARCore support it silently won't, which is a crash waiting to happen if you skip the runtime check.

So the first step is the check itself — ARCore publishes a list of supported devices, and the right pattern is to query the availability at runtime and degrade gracefully:

if (ArCoreApk.getInstance().checkAvailability(context) ==
    ArCoreApk.Availability.SUPPORTED_NOT_INSTALLED
) {
    ArCoreApk.getInstance().requestInstall(this, true)
}

You also need to know the minimum hardware bar for your own QA: a device with a decent camera and a gyroscope. The shelf-in-the-room demo was fine on the test devices, and it was fine in the store demo — because the demo devices were exactly the ones I certified. That is the first lesson: certify a small device list before you promise the feature to everyone.

Step 2: Add Dependencies and the Manifest

Two pieces of infrastructure do the heavy lifting. ARCore provides the session, plane detection, and hit testing; SceneView wraps the Filament renderer so you get realistic lighting, shadows, and occlusion without writing a renderer by hand.

dependencies {
    implementation("io.github.sceneview:arsceneview:2.3.0")
    implementation("com.google.ar:core:1.45.0")
    implementation("com.google.android.filament:filament-android:1.52.1")
}

Then the manifest. Three things matter and each one has caused a "works on my machine, crashes in QA" bug:

<manifest>
    <!-- 1. Camera permission -->
    <uses-permission android:name="android.permission.CAMERA" />

    <!-- 2. ARCore required for devices that have it -->
    <meta-data
        android:name="com.google.ar.core"
        android:value="required" />

    <!-- 3. GLES 2.0+ is the minimum for Filament -->
    <uses-feature android:glEsVersion="0x00020000" android:required="true" />
</manifest>

Two notes. First, android:value="required" means the Play Store will refuse to install the app on devices without ARCore — which is what you want for a feature that is the app's reason to exist, but it will silently shrink your reach. If AR is a secondary feature, use "optional" and gate it at runtime instead. Second, request the camera permission in code with ActivityResultContracts.RequestPermission() before you start the session — ARCore throws the moment it cannot open the camera.

Step 3: Set Up the ARCore Session

The session is the engine that tracks the world — the camera pose, the surfaces it finds, the anchors it holds. In practice you rarely manage it by hand with SceneView, because the wrapper starts a session for you, but you need to know what is happening under the hood: the session builds a 3D understanding of the scene from camera frames plus the device's motion sensors, and it exposes "planes" — flat surfaces it is increasingly confident about.

class ArActivity : ComponentActivity() {
    private lateinit var sceneView: ArSceneView

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_ar)
        sceneView = findViewById(R.id.sceneView)
        sceneView.planeRenderer.isVisible = true
    }
}

planeRenderer.isVisible = true shows the wireframe grid so the user understands the feature is "looking" for a floor. In the furniture demo this single visual detail halved the "why is it not working" confusion — AR features need affordance as much as any other UI, and an invisible detector looks like a bug.

Step 4: Load a 3D Model

For a real product you need an actual model. A .glb file is the asset format that Filament consumes — if your designer exports from Blender or Cinema 4D, the glTF pipeline is what you want. Model scale is where every AR demo goes wrong: a shelf modeled in Blender is in meters, but the units in your scene are whatever the model says. If the model was authored in centimeters, your "shelf" will be a hundred times too big or the size of a grain of rice.

// In onSurfaceReady, after the session starts:
sceneView.onSurfaceReady { view ->
    val model = ModelData.create(view.context, R.raw.shelf_glb)
    view.setModel(model)
}

Get the asset into res/raw/, and check the model's scale on the first render, not the twentieth. The pattern that saves hours: place the model once, put a real-world object next to it on screen (a bottle, a hand), and adjust the scale until the size reads true.

Step 5: Tap-to-Place with a Hit Test

This is the core interaction, and it is two operations glued together. A tap is translated into a hit test — a ray from the camera through the screen point into the tracked world — and ARCore returns a HitResult for any plane that ray crossed. You anchor the model to that point so it stays glued to the world as the user moves the phone.

sceneView.setOnTapListener { hitResult ->
    // ARCore returns the point on the detected plane, not a raw screen coord.
    val node = ModelNode(
        modelData = ModelData.create(this, R.raw.shelf_glb),
        parentNode = sceneView,
        anchor = hitResult.createAnchor()
    )
    node.scaleTo(0.6f) // scale relative to the anchor
    sceneView.addChild(node)
}

The createAnchor() line is the whole trick. An anchor fixes a position in real space; without it, the model floats relative to the camera and slides around as you move, which is the #1 "AR feels fake" symptom. With the anchor, the shelf sits on the floor and stays there while the user walks around it. Note the hit test only returns meaningful results when the session has actually detected a plane — on a plain white wall or a low-texture floor, the hit test returns nothing, which brings us to the next step.

Model Prep: The Step Everyone Skips

A working placement is not a shippable model. The asset pipeline decides whether your feature feels polished or janky, and it is the step most tutorials skip. Three rules:

• Author with physically based materials. Filament renders metallic/roughness textures realistically, and the occlusion lighting in the next step only looks right when materials are physically based. A flat-textured model will look like a sticker pasted into a real room, no matter how good the lighting pass is.
• Export to glTF and compress. The .glb you ship should be a few megabytes at most. If your designer hands you a 120 MB Blender export, it will choke low-end devices at load time. Run the file through the glTF pipeline, strip unused textures, and time the load on your slowest certified device.
• Provide a low-poly fallback. AR places objects at arbitrary scales; a model that looks fine at one meter looks blocky at four. Ship a lower-detail variant for large placements and swap based on the anchor scale.

Model prep is the source of 80 percent of the "why does it look cheap" complaints I have collected, and it has nothing to do with ARCore.

Step 6: Occlusion and Lighting (The Realism Layer)

An AR feature that looks like a floating hologram fails the moment a customer sees it. Two mechanisms in this stack fix that, and they are the difference between a tech demo and a product:

Occlusion. The model must disappear behind real objects — walk behind a chair and the shelf should hide, not hover through the chair. ARCore's Depth API provides a depth map of the scene, and SceneView uses it so the Filament renderer clips the model correctly against real geometry. Enable it and the feature stops looking like a green-screen insert.

Lighting. Filament doesn't just render the model; it reads the ambient lighting estimate from the session and lights the model accordingly. A model lit for a bright studio looks wrong in a dim living room. Let the renderer use the estimate, keep the model's own materials physically based (metallic/roughness), and the shelf picks up the room's actual shadows and tones. This is the single highest-visual-impact step in the whole pipeline, and it is one flag plus a physically-based-material asset — no renderer work required.

Step 7: Handle the Failure Modes in Code

You will hit these; plan for them before they hit you:

• ARCore not supported. The device list is finite. Route to a fallback screen ("Your device supports this") instead of a crash.
• Camera permission denied. The session throws. Check the permission result before starting the session, and re-prompt with an explanation.
• No plane detected. On low-texture surfaces the session takes seconds or never finds a plane. Show a "move your phone slowly" hint — the plane renderer helps here.
• Session loss. Point the camera at a blank wall and tracking can drop. Recover by restarting the session, not by restarting the activity.
• Lifecycle. ARCore sessions consume the camera and the GPU. Pause the session in onPause, resume in onResume, and release in onDestroy or the camera stays locked for the next app that needs it.

Performance: The 60 FPS Budget

AR is the most performance-sensitive UI you will ship, because the camera feed plus the 3D scene must hold 60 frames per second or the illusion collapses into queasiness. The budget is fixed, and the usual culprits are predictable:

• The session and the renderer run on the main thread by default. Any main-thread work — a network call, a synchronous decode, an un-optimized layout — drops frames. Offload everything that is not rendering.
• Fill rate kills low-end devices. Too many draw calls or full-screen translucent overlays tank older GPUs. Keep the plane visualization simple, batch the model geometry, and test on the low end, not your flagship.
• Battery and heat are real UX features. A camera session at 60fps drains a battery fast. Track session length, lower the frame rate when the app backgrounds, and let the user pause the AR session; nothing converts worse than a hot phone.

Profile with the Android GPU Profiler, not intuition. In my first AR build, the model was fine — the dropped frames came from an unrelated main-thread image decode on the same screen.

Step 8: Test on Real Devices, Not Just the Demo Phone

AR is the one Android feature where "it works on my device" is genuinely not good enough. A functional test on your flagship is not a certification. Run the flow on the cheapest ARCore-supported phone you can find — the low-end device is where the plane detection is slowest, the tracking is jankiest, and the session loss is most frequent. If it holds on the low end, it is ready for production. Also test in two lighting conditions (a bright room and a dim one) and on at least one textured floor and one plain one; those four combinations catch most of the "it worked in the demo" failures.

When AR Is the Wrong Feature

Be honest about whether your product needs this at all. AR is expensive — in device compatibility, in testing matrix, in performance budget — and a feature nobody uses is a tax, not a differentiator. The furniture client's shelf demo earned its keep because placement anxiety is a genuine purchase blocker in that category. But I have watched brands bolt an AR viewer onto products where a good 3D rotation or a clear photo gallery answers the same question at a fraction of the cost. The decision rule: AR earns its budget when spatial uncertainty is the reason customers hesitate. If a customer can buy confidently from a photo, give them a photo. If scale, fit, or placement is the blocker, that is the moment AR pays.

The Practitioner's Checklist

• Runtime ARCore availability check with graceful fallback
• Camera permission requested and denied-path handled
• Manifest: camera permission, ARCore meta-data, GLES minimum
• .glb asset authored in real-world units, scale verified against a real object
• Anchors on every placed object — no floating models
• Occlusion enabled (depth API) so objects hide behind real geometry
• Ambient lighting estimate used; physically-based materials
• Session paused/resumed in lifecycle, camera released on destroy
• Tested on a low-end ARCore device and in dim + bright lighting
• "Move your phone slowly" hint for slow plane detection

The Honest Postscript

The shelf feature shipped, and the week it took split cleanly into two halves: the first half was plumbing — session, manifest, permissions, the model that was the wrong size — and the second half was the realism layer that made the feature feel like a product. Everything in this stack is genuinely approachable: ARCore handles the hard tracking math, SceneView wraps Filament's excellent renderer, and your job reduces to the parts that make a feature feel honest — anchors, occlusion, lighting, and testing on the hardware your customers actually have. Start with the manifest, place one model, and let the Depth API do the rest.

*Gulshan Yad

Advanced AR Feature Development Techniques

Advanced AR feature development techniques include implementing advanced gestures and interactions, such as pinch-to-zoom and drag-to-rotate. This can be achieved using the ARFragment class and implementing custom gestures using the GestureDetector class.

Optimizing 3D Models for AR Feature Development

To optimize 3D models for AR feature development, use the Sceneform library to ensure they are in a supported format. This includes using the correct texture sizes and formats, and optimizing the 3D model for mobile devices.

Implementing Plane Detection and Tracking

Plane detection and tracking is a critical component of AR feature development. Use the ARCore SDK to implement plane detection and tracking, and handle user input accordingly. This includes detecting planes and tracking their movement over time.

Adding Real-World Interactions

Adding real-world interactions to an AR experience can enhance the user's engagement and interaction with the virtual objects. Use the ARFragment class and implement custom gestures using the GestureDetector class to achieve this.

Advanced Sceneform Techniques

Advanced Sceneform techniques include using the Sceneform library to create complex 3D scenes, and implementing custom shaders to enhance the visual appearance of the 3D models. This can be achieved by using the Sceneform library's built-in features, such as the Material class and the Lighting class.

Testing and Debugging AR Experiences

Testing and debugging AR experiences is critical to ensure they work correctly and perform well on a variety of devices. Use the Android Studio debugger and the ARCore SDK's built-in debugging tools to identify and fix issues.

Key Takeaways

• Create a new Android project and set up the necessary dependencies for AR feature development.
• Implement the ARCore SDK and set up the necessary permissions for AR feature functionality.
• Use the ARFragment class to display the AR experience and handle user input.
• Load and display 3D models using the Sceneform library.
• Implement plane detection and tracking using the ARCore SDK.
• Add gestures and interactions to enhance the AR experience.