Scientists at the Massachusetts Institute of Technology (MIT CSAIL) have developed an innovative system for controlling robots — " Neural Jacobian Fields "(NJF), which radically changes the approach to teaching robotics. The system allows robots of any type and complexity to learn how to control their bodies independently, using only visual data from the camera, without using complex sensors or pre-defined models.
Traditional methods require precise programming and multiple sensors to create a digital robot model. This complicates the development of soft, deformable or non-standard robots. NJF changes the rules of the game: the robot "observes" its own random movements using a camera, and forms an internal model of its body based on them. This learning process can be compared to how a person learns to control finger movements - through practice and feedback.
NJF is based on a neural network that links the three-dimensional shape of a robot with its response to control commands. The system uses the neural Radiation fields (NeRF) method, extended to train the Jacobian field, a mathematical model that describes how any point in the robot's body moves in response to motor commands. Thanks to this, the robot is able to predict and correct its movements in real time, using only the video stream from one camera.
Testing has shown high efficiency of NJF on a variety of robots: pneumatic soft arm, Allegro rigid arm, 3D-printed shoulder and rotating platform without built-in sensors. In all cases, the system independently studied the shape and dynamics of devices, demonstrating accuracy and learning speed that surpasses traditional physical simulators, which often turn out to be too resource-intensive.
The potential of the technology is huge: robots with NJF will be able to perform precise agricultural work, work on construction sites without complex equipment, and adapt to rapidly changing environmental conditions. Currently, training requires multiple cameras and repeating the procedure for each robot, but researchers are already working on a simplified version that will allow you to use conventional smartphones to record movements and create control models without specialized equipment and deep technical knowledge.
