AgentHands: Google gives XR AI agents the ability to point out where things are

Current AI assistants can already see fairly well through a camera. Tools like Project Astra or Gemini Live can react to their surroundings and often use 2D bounding boxes to highlight what they're talking about. But on platforms like Android XR, this flat UI approach isn't enough. That's why Google is releasing AgentHands - a research prototype from the CHI 2026 conference that brings full 3D non-verbal communication back to agents.

The end of flat UI in spatial environments

In Extended Reality, you don't need colored boxes around objects. You need the agent to naturally indicate a shape, suggest an action, or emphasize an important point - all perfectly synchronized with its voice. AgentHands is an LLM-powered system that does exactly this. It turns abstract instructions into clear visual demonstrations in your environment.

Interaction shifts from text to physical orientation

This shift marks the end of trying to describe 3D objects using 2D words. A virtual hand that can point to exactly where to start cutting material or which port to plug a cable into is far more useful in some industrial and educational scenarios than a long explanatory text or a static image.

An agent with manual navigation capabilities

For AR and XR glasses users, this fundamentally changes interaction ergonomics. When you ask an agent how to fix something or where something is, you no longer have to decode interface overlays. Virtual hands point directly to a specific screw or demonstrate the exact gesture to move it. Google focused heavily on gesture legibility, creating a specific taxonomy for it (Handedness, Gesture, Spatiality, Temporal Dynamics, Interactivity, Visual Effects).

Improved orientation will bring XR to the workplace

The real proof of this approach's viability won't be a lab demo, but whether these synchronized 3D layers can be reliably generated in real-time for open-ended tasks without choking on complex spatial instructions. If it works smoothly, we're looking at the new standard for how agents in AR glasses will communicate with our physical world.

Lilith's verdict

No more dumb red rectangles like on a security camera dashboard. This is the step where the assistant stops being a layer over a display and becomes a real actor in the room.