Brain waves may unlock the next breakthrough in physical AI training

Encord, a company specializing in data tools for AI training, is testing brain wave headsets from German startup Zander Labs to enhance the quality of robotic training data. The trial involves pilots wearing headsets that track both visual input and brain activity while performing tasks like pulling blocks from a Jenga tower or pouring coffee.

The key hypothesis is that measuring brain activity can reveal mental states such as error detection, intention, and surprise, which can be annotated to create richer datasets. Zander neuroscientist Lucas Gehrke notes that the amount of brain activity at different points of a task tells model builders when to deploy more complex processing.

This effort is part of Encord's broader push to manufacture physical training data rather than just manage it. According to Vineeth Velmurugan, head of robot learning at Encord, the scarcity of real-world data is the biggest bottleneck in physical AI. He estimates that a dataset roughly five times larger than YouTube's entire video corpus is needed to break through current limitations.

Encord collects data through multiple modalities: egocentric video from workers wearing cameras in factories around the world, leader-follower robotic rigs for precise manipulation tasks, and experimental sensors like brain wave monitors and forearm muscle sensors. The arm sensors detect electrical signals in muscles to build a 3D model of hand position, addressing the limitations of standard video.

Data is annotated with detailed descriptions like "right hand tightens bolt," which Velmurugan says can be 100 times more valuable for training specific tasks than raw ego data, yet costs only 20 times more to produce. However, this cost structure contrasts sharply with LLM training, where internet text is essentially free.

Encord's pilots, including Sofia Infante and Andrew Ceja, come from backgrounds in AI data annotation and waste management robotics, respectively. They work on tasks ranging from plugging ethernet cables to stacking poker chips, generating data that top robotics companies need.

Since its founding, Encord has evolved from a data annotation service to a data creation company, responding to customer demand for end-to-end learning in robotic manipulation. The company's cross-industry visibility allows it to identify emerging data techniques before any single client can, keeping its San Leandro facility busy with experiments like brain wave tagging.