Spatial Lingo: An Open Source App for AI-Assisted Language Practice with Everyday Objects

What if your normal surroundings could teach you a new language?

With our new open source app Spatial Lingo⁠, everyday objects become labeled in your choice of English or Spanish through mixed reality to make practicing a new language immersive and fun. Keep reading to see how this open source Unity project connects Passthrough, AI, and mixed reality UI together (and how to extend it to your own projects).

Spatial Lingo is an open source mixed reality app that reimagines language practice by adding vocabulary in the language of your choice to the world around you. You can try Spatial Lingo⁠ in the Meta Horizon Store (U.S. only) today and check out the open source project⁠, which is available for developers everywhere, to see how it works. Spatial Lingo leverages Passthrough Camera API and multiple AI technologies to deliver a great foundation for implementing these capabilities into your own experiences.

Instead of memorizing vocabulary from a list, users learn new words by exploring their real environment—objects you see every day are instantly translated and highlighted to make the process intuitive and memorable. Whether you’re at home, in the office, or out and about, Spatial Lingo turns your surroundings into an interactive classroom. The app’s playful 3D companion Golly Gosh guides you through the experience, offering encouragement and making learning feel less like a chore and more like an adventure.

Spatial Lingo offers a prime example of what’s possible on Meta Quest when the Passthrough Camera API and multiple AI technologies come together.

Using Passthrough Camera API, Spatial Lingo can see what’s around the player in their physical space. Using YOLOv9 running through Unity’s Inference Engine (aka Sentis) to analyze the images from the Quest’s cameras, the app accurately identifies common objects around the player.

Once an object is identified with YOLO, the image is cropped and sent to Llama API, which analyzes the object using Llama 4 Maverick to obtain accurate nouns, adjectives, and verbs based on that specific object. This functionality provides a dynamic experience that feels inherently tailored to the user by highlighting objects they can see and touch in front of them.

By default, Spatial Lingo uses Llama 4 Maverick (through the Llama API) to translate the names of detected objects into Spanish and render them in mixed reality. You can also swap in your own model and AI API for translations using the open source project on GitHub⁠.

When the player is prompted to speak, the app uses Voice SDK to capture and transcribe their voice input. That transcript is then sent to Llama 4 Maverick, which checks whether the player is using the target words correctly—down to conjugation and basic sentence structure.

Golly Gosh’s dialogue is also generated by Llama 4 Maverick (again, accessed via Llama API), which lets the character respond dynamically to what’s happening in the session. Once the text response is generated, Voice SDK converts it to speech so Golly Gosh can respond in an encouraging and lighthearted voice.

Spatial Lingo is open source, and we invite developers to try it out, contribute, and build on top of our work.

Spatial Lingo offers a practical blueprint for building mixed reality apps with AI and Passthrough. Dive in today and check out the app to get inspired, view the source code as an example, or even clone the project and build a new app on top of it.

Want more developer news? Check out the Meta Horizon Developer Forum⁠, subscribe to our monthly newsletter,⁠ and follow us on X⁠ and Facebook⁠.