Inside the Augmented Reality Mirror: How Xtendr Builds Immersive AR Experiences

When did you first hear the term augmented reality (AR)? If you’re new to the tech space, your first encounter with this visual interactive technology was probably the Apple Vision Pro or Meta Quest headset. However, gamers would most likely interact with the technology for the first time in the mobile game Pokemon Go, which was released in 2016. Movie buffs may also refer to the 2008 movie Iron Man, where JARVIS gave a glimpse of what it can to be the future of AR.

Interestingly, the origin of this technology predates all these modern stages. The first AR device was created in 1968, when American computer scientist Ivan Sutherland invented a head-mounted display that could put two-dimensional images into a 3D environment. It was a massive heavy device that was suspended from the ceiling and only showed rudimentary rooms with a wire structure. However, this invention became the foundation of AR for decades to come.

What is Augmented Reality (AR)

One of the most interesting technologies in the modern world, AR is a conceptually straightforward mechanism. It creates a 2D or 3D overlay of digital information on a real-world environment. It basically shows you information, objects, animation or an entire visual scene in the middle of your environment. For example, the game Pokemon Go can place moving Pokemon on your table or on the floor as long as you point your camera at it.

Apple Vision Pro and Meta Quest have expanded the possibilities of AR. While viewing objects or a movie with this effect is a great new experience, the real immersion comes when the user is not only able to view digital information in the real world, but is also able to interact with it. And mixed reality headsets allow for just that. People using these headphones can watch a movie, play a game, or use an app, which is essentially a floating screen in the middle of your room. It can be expanded or minimized according to your preference.

This growing understanding of augmented reality and the development of hardware infrastructure to support such technology today ensures that these experiences are not limited to certain applications or high-end devices, but are all around us to be experienced in a platform-free environment. Today, multiple Indian companies are creating AR content that can be experienced right on the smartphone with a click of a button on a website or by scanning a QR code — without the need to buy or download anything.

Some big names in the space include Imaginate, Parallax Labs, PlayShifu, Xtendr, and more. Here’s an example to put it in the context of what exactly they’re building. If you’re a fan of Maggi, you might remember that the company recently launched its Korean BBQ noodles where the packaging had a visible QR code. Scanning it will open an AR session, allowing you to cook noodles in a 3D interactive space. The experience acted as a how-to for those unfamiliar with the product and allowed people to play a mini-game without having to download or install anything.

This particular platform-less AR experience was created by Xtendr, and there are a lot of little things behind creating such experiences. To understand the technology behind such AR technology and its potential, we at Gadgets 360 had a conversation with Anurag Sachdeva, founder and CEO of the company.

Xtendr’s foray into platform-less AR

The vision of building a company focused on AR is not common. But for Sachdeva, that path was paved by his vast 23 years of experience with technology-focused companies. Before trying his hand at entrepreneurship, he headed Rovio Entertainment’s Angry Birds business for India, Southeast Asia and the Middle East. The company is getting quite involved with AR; however, everything was managed through the apps.

As a marketing professional, Sachdeva wanted to take advantage of the engagement that comes with an immersive AR experience, but without the limitations that an app brings. This idea led to the founding of Xtendr in May 2020. However, the road was not without its challenges. Launching a company in the middle of a global pandemic meant making adjustments to keep it going.

“We were the first in the region to create AR experiences in a way that did not require an app. In the early days, one concern was understanding our target demographic at a time when people were also dealing with the COVID-19 pandemic,” said Sachdeva. Despite initial hiccups, the startup was able to attract two brands, and the founder claims they’ve seen great results in terms of engagement. This development gave Xtendr the momentum it needed.

Today, Xtendr works with more than 50 brands across India, the Middle East and Southeast Asia. Some of the brands he has worked with include Redbull, Coca-Cola, ISRO, Unilever, Nestle Maggi, ITC Classmate, etc. But how exactly do these “platform-less AR experiences” work? We dug deeper.

Breaking down Xtendr’s AR experiences

Before we jump in, let’s take a moment to understand what Xtendr does. The startup works with consumer brands and other organizations and creates branded content in the AR space. They can be anything from a small animation that plays on your bed to a fully interactive game of cricket that you can play in your garden. Once activated via a URL or QR code, it loads onto your device without the need to download an app. These experiences, also known as web AR, run entirely in the cloud.

However, building such experiences requires multiple components to work properly. Latency should be low so that these experiences don’t take forever to load on devices with a poor internet connection. Experiences must be fully AR supported, immersive and interactive.

Sachdeva told us that Xtendr uses spatial computing and artificial intelligence (AI) to create augmented reality. Immersion is a major focus for the company, which aims to “expand human vision.”

The actual creation process involves first finding an idea and creating a narrative-driven narrative through characters, objects, and elements. Once the storyboard is ready, the technical hard work begins in the back end. The content is then converted into a program that can be run seamlessly on the end user’s device.

The company primarily uses JavaScript to write the programs using the Three.js library. Web AR and VR frameworks such as A-Frame, SLAM (Simultaneous Localization and Mapping), and React VR are also being used in development. All of this goes to the cloud where the front end comes together and the AR experience is ready for distribution.

The role of AI

Interestingly, the startup is also incorporating generative AI into some AI experiences. The company uses large-scale language models (LLM) for characters when they want to give users a conversational experience. AI is also used to generate art and allow users to enjoy visual media generated by AI.

These tools are used to design a gamified experience where the user can talk to a chatbot about the brand or do weird things like draw the sky. Some AI models that Xtendr works with include Inworld, Midjourney, ChatGPT, Gemini, Dall-E, and more.

Fix the lag problem

A key part of creating a platform-free AR experience is its distribution. Users would prefer their content to load in seconds and ensuring this from budget to premium devices and from low to excellent internet speed is a big challenge. This becomes important, especially as experiences include 3D characters, complex artwork, and interactive elements.

Xtendr has created game engagements in the past where users can play a game and challenge another player online or on the leaderboard where many others also post their scores. With one AR experience reaching two million impressions, it becomes important to ensure consistency.

The startup uses Amazon Web Services (AWS) for its cloud-based needs, which solves part of the problem, as the cloud computing giant handles some of the most traffic-heavy platforms, such as Netflix, Twitch, LinkedIn, Facebook, and more. The other part is solved by optimizing the 3D renderings to ensure that none are too heavy for any device. Sachdeva emphasized the need for a balance between art and interaction, which contribute equally to engagement.

A look at the future of augmented reality

For a technology that first arrived 56 years ago, AR remains in its infancy largely due to the lack of technology that can support the vision. But in the last few years it has really entered the mainstream and become viable and profitable. That’s why companies are experimenting with it more and more. Today, AR has evolved from a party trick to a useful tool. Digital overlay on human vision opens up new avenues for productivity online while still connected to the real world instead of glued to a screen.

Sachdeva argues that the future of AR is bright as all the key elements needed for the technology to grow exponentially are finally coming together. “5G will become mainstream in the next few years, which will further solve many of the latency issues. AR-enabled hardware, high processing power, the emergence of generative AI, and the advent of mixed reality headsets [by Apple and Meta]everything is coming together to create the next highway for immersive media,” he added.